2026-10-27 17:29:33,402 [ 99500 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy (runner:42, check_args_and_update_paths)
2026-10-27 17:29:33,402 [ 99500 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:90, check_args_and_update_paths)
2026-10-27 17:29:33,402 [ 99500 ] INFO : src dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/src (runner:97, check_args_and_update_paths)
2026-10-27 17:29:33,402 [ 99500 ] INFO : base_configs_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration  (runner:99, check_args_and_update_paths)
2026-10-27 17:29:33,402 [ 99500 ] INFO : Unknown image altinityinfra/dotnet-client (runner:357, <module>)
Running pytest container as: 'docker run   --rm --name clickhouse_integration_tests_wa6pwn --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=1864 -e DOCKER_BASE_TAG=1864 -e DOCKER_KERBERIZED_HADOOP_TAG=1864 -e DOCKER_KERBEROS_KDC_TAG=1864 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=1864 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=1864 -e DOCKER_MYSQL_JS_CLIENT_TAG=1864 -e DOCKER_MYSQL_PHP_CLIENT_TAG=1864 -e DOCKER_NGINX_DAV_TAG=1864 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=1864   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600  -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_storage_azure_blob_storage/test.py::test_partition_by_const_column test_storage_azure_blob_storage/test.py::test_partition_by_string_column test_storage_azure_blob_storage/test.py::test_partition_by_tf test_storage_azure_blob_storage/test.py::test_put_get_with_globs test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container test_storage_azure_blob_storage/test.py::test_read_subcolumns test_storage_azure_blob_storage/test.py::test_schema_inference_cache test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf test_storage_azure_blob_storage/test.py::test_simple_read_write test_storage_azure_blob_storage/test.py::test_simple_write_account_string test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function test_storage_azure_blob_storage/test.py::test_simple_write_connection_string test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1 test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2 test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function 'test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]' 'test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]' test_storage_azure_blob_storage/test.py::test_truncate test_storage_hdfs/test.py::test_bad_hdfs_uri test_storage_hdfs/test.py::test_cluster_join test_storage_hdfs/test.py::test_cluster_macro test_storage_hdfs/test.py::test_format_detection test_storage_hdfs/test.py::test_globs_in_read_table test_storage_hdfs/test.py::test_hdfsCluster test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards test_storage_hdfs/test.py::test_hdfs_directory_not_exist test_storage_hdfs/test.py::test_insert_select_schema_inference test_storage_hdfs/test.py::test_multiple_inserts test_storage_hdfs/test.py::test_overwrite test_storage_hdfs/test.py::test_partition_by test_storage_hdfs/test.py::test_read_files_with_spaces test_storage_hdfs/test.py::test_read_subcolumns test_storage_hdfs/test.py::test_read_table_with_default test_storage_hdfs/test.py::test_read_write_gzip_table test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip test_storage_hdfs/test.py::test_read_write_storage test_storage_hdfs/test.py::test_read_write_storage_with_globs test_storage_hdfs/test.py::test_read_write_table test_storage_hdfs/test.py::test_read_write_table_with_parameter_none test_storage_hdfs/test.py::test_schema_inference test_storage_hdfs/test.py::test_schema_inference_cache test_storage_hdfs/test.py::test_schema_inference_with_globs test_storage_hdfs/test.py::test_seekable_formats test_storage_hdfs/test.py::test_skip_empty_files test_storage_hdfs/test.py::test_storage_with_multidirectory_glob test_storage_hdfs/test.py::test_truncate_table test_storage_hdfs/test.py::test_virtual_columns test_storage_hdfs/test.py::test_virtual_columns_2 test_storage_hdfs/test.py::test_write_gz_storage test_storage_hdfs/test.py::test_write_gzip_storage test_storage_hdfs/test.py::test_write_table test_storage_kerberized_hdfs/test.py::test_cache_path test_storage_kerberized_hdfs/test.py::test_prohibited test_storage_kerberized_hdfs/test.py::test_read_table test_storage_kerberized_hdfs/test.py::test_read_table_expired test_storage_kerberized_hdfs/test.py::test_read_write_storage test_storage_kerberized_hdfs/test.py::test_two_users test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials 'test_storage_url/test.py::test_file_formats[CSV]' 'test_storage_url/test.py::test_file_formats[JSONEachRow]' 'test_storage_url/test.py::test_file_formats[Parquet]' 'test_storage_url/test.py::test_file_formats[TSV]' test_storage_url/test.py::test_partition_by test_storage_url/test.py::test_table_function_url_access_rights test_storage_url/test.py::test_url_cluster test_storage_url/test.py::test_url_cluster_with_named_collection test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric test_system_metrics/test.py::test_broken_tables_readonly_metric test_system_metrics/test.py::test_metrics_storage_buffer_size test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name test_wrong_db_or_table_name/test.py::test_wrong_database_name test_wrong_db_or_table_name/test.py::test_wrong_table_name test_zero_copy_fetch/test.py::test_concurrent_move_to_s3 test_zero_copy_fetch/test.py::test_fetch_correct_volume test_zero_copy_fetch/test.py::test_zero_copy_mutation test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part test_zookeeper_config/test_password.py::test_identity  -vvv" altinityinfra/integration-tests-runner:1864 '.
Start tests
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.7.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /ClickHouse/tests/integration
configfile: pytest.ini
plugins: repeat-0.9.4, order-1.0.1, random-0.2, timeout-2.2.0, xdist-3.5.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
created: 10/10 workers
10 workers [90 items]

scheduling tests via LoadFileScheduling

test_storage_hdfs/test.py::test_bad_hdfs_uri 
test_storage_azure_blob_storage/test.py::test_partition_by_const_column 
test_storage_url/test.py::test_file_formats[CSV] 
test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric 
test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance 
test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name 
test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part 
test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections 
test_zero_copy_fetch/test.py::test_concurrent_move_to_s3 
test_storage_kerberized_hdfs/test.py::test_cache_path 
[gw8] [  1%] PASSED test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance 
test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance 
[gw8] [  2%] PASSED test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance 
[gw4] [  3%] PASSED test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name 
test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name 
[gw4] [  4%] PASSED test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name 
test_wrong_db_or_table_name/test.py::test_wrong_database_name 
[gw4] [  5%] PASSED test_wrong_db_or_table_name/test.py::test_wrong_database_name 
test_wrong_db_or_table_name/test.py::test_wrong_table_name 
[gw4] [  6%] PASSED test_wrong_db_or_table_name/test.py::test_wrong_table_name 
[gw2] [  7%] PASSED test_storage_url/test.py::test_file_formats[CSV] 
test_storage_url/test.py::test_file_formats[JSONEachRow] 
[gw2] [  8%] PASSED test_storage_url/test.py::test_file_formats[JSONEachRow] 
test_storage_url/test.py::test_file_formats[Parquet] 
[gw2] [ 10%] PASSED test_storage_url/test.py::test_file_formats[Parquet] 
test_storage_url/test.py::test_file_formats[TSV] 
[gw2] [ 11%] PASSED test_storage_url/test.py::test_file_formats[TSV] 
test_storage_url/test.py::test_partition_by 
[gw2] [ 12%] PASSED test_storage_url/test.py::test_partition_by 
test_storage_url/test.py::test_table_function_url_access_rights 
[gw2] [ 13%] PASSED test_storage_url/test.py::test_table_function_url_access_rights 
test_storage_url/test.py::test_url_cluster 
[gw2] [ 14%] PASSED test_storage_url/test.py::test_url_cluster 
test_storage_url/test.py::test_url_cluster_with_named_collection 
[gw5] [ 15%] PASSED test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric 
test_system_metrics/test.py::test_broken_tables_readonly_metric 
[gw2] [ 16%] PASSED test_storage_url/test.py::test_url_cluster_with_named_collection 
[gw5] [ 17%] PASSED test_system_metrics/test.py::test_broken_tables_readonly_metric 
test_system_metrics/test.py::test_metrics_storage_buffer_size 
[gw5] [ 18%] PASSED test_system_metrics/test.py::test_metrics_storage_buffer_size 
[gw9] [ 20%] PASSED test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part 
[gw6] [ 21%] PASSED test_zero_copy_fetch/test.py::test_concurrent_move_to_s3 
test_zero_copy_fetch/test.py::test_fetch_correct_volume 
[gw6] [ 22%] PASSED test_zero_copy_fetch/test.py::test_fetch_correct_volume 
test_zero_copy_fetch/test.py::test_zero_copy_mutation 
[gw6] [ 23%] PASSED test_zero_copy_fetch/test.py::test_zero_copy_mutation 
[gw3] [ 24%] PASSED test_storage_kerberized_hdfs/test.py::test_cache_path 
test_storage_kerberized_hdfs/test.py::test_prohibited 
[gw3] [ 25%] PASSED test_storage_kerberized_hdfs/test.py::test_prohibited 
test_storage_kerberized_hdfs/test.py::test_read_table 
[gw1] [ 26%] PASSED test_storage_azure_blob_storage/test.py::test_partition_by_const_column 
test_storage_azure_blob_storage/test.py::test_partition_by_string_column 
[gw1] [ 27%] PASSED test_storage_azure_blob_storage/test.py::test_partition_by_string_column 
test_storage_azure_blob_storage/test.py::test_partition_by_tf 
[gw1] [ 28%] PASSED test_storage_azure_blob_storage/test.py::test_partition_by_tf 
test_storage_azure_blob_storage/test.py::test_put_get_with_globs 
[gw3] [ 30%] PASSED test_storage_kerberized_hdfs/test.py::test_read_table 
test_storage_kerberized_hdfs/test.py::test_read_table_expired 
[gw1] [ 31%] PASSED test_storage_azure_blob_storage/test.py::test_put_get_with_globs 
test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf 
[gw0] [ 32%] PASSED test_storage_hdfs/test.py::test_bad_hdfs_uri 
test_storage_hdfs/test.py::test_cluster_join 
[gw0] [ 33%] PASSED test_storage_hdfs/test.py::test_cluster_join 
test_storage_hdfs/test.py::test_cluster_macro 
[gw0] [ 34%] PASSED test_storage_hdfs/test.py::test_cluster_macro 
test_storage_hdfs/test.py::test_format_detection 
[gw0] [ 35%] PASSED test_storage_hdfs/test.py::test_format_detection 
test_storage_hdfs/test.py::test_globs_in_read_table 
[gw1] [ 36%] PASSED test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf 
test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container 
[gw1] [ 37%] PASSED test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container 
test_storage_azure_blob_storage/test.py::test_read_subcolumns 
[gw1] [ 38%] PASSED test_storage_azure_blob_storage/test.py::test_read_subcolumns 
test_storage_azure_blob_storage/test.py::test_schema_inference_cache 
[gw0] [ 40%] PASSED test_storage_hdfs/test.py::test_globs_in_read_table 
test_storage_hdfs/test.py::test_hdfsCluster 
[gw0] [ 41%] PASSED test_storage_hdfs/test.py::test_hdfsCluster 
test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards 
[gw0] [ 42%] PASSED test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards 
test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards 
[gw0] [ 43%] PASSED test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards 
test_storage_hdfs/test.py::test_hdfs_directory_not_exist 
[gw0] [ 44%] PASSED test_storage_hdfs/test.py::test_hdfs_directory_not_exist 
test_storage_hdfs/test.py::test_insert_select_schema_inference 
[gw0] [ 45%] PASSED test_storage_hdfs/test.py::test_insert_select_schema_inference 
test_storage_hdfs/test.py::test_multiple_inserts 
[gw0] [ 46%] PASSED test_storage_hdfs/test.py::test_multiple_inserts 
test_storage_hdfs/test.py::test_overwrite 
[gw0] [ 47%] PASSED test_storage_hdfs/test.py::test_overwrite 
test_storage_hdfs/test.py::test_partition_by 
[gw0] [ 48%] PASSED test_storage_hdfs/test.py::test_partition_by 
test_storage_hdfs/test.py::test_read_files_with_spaces 
[gw0] [ 50%] PASSED test_storage_hdfs/test.py::test_read_files_with_spaces 
test_storage_hdfs/test.py::test_read_subcolumns 
[gw0] [ 51%] PASSED test_storage_hdfs/test.py::test_read_subcolumns 
test_storage_hdfs/test.py::test_read_table_with_default 
[gw0] [ 52%] PASSED test_storage_hdfs/test.py::test_read_table_with_default 
test_storage_hdfs/test.py::test_read_write_gzip_table 
[gw0] [ 53%] PASSED test_storage_hdfs/test.py::test_read_write_gzip_table 
test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz 
[gw0] [ 54%] PASSED test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz 
test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip 
[gw0] [ 55%] PASSED test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip 
test_storage_hdfs/test.py::test_read_write_storage 
[gw0] [ 56%] PASSED test_storage_hdfs/test.py::test_read_write_storage 
test_storage_hdfs/test.py::test_read_write_storage_with_globs 
[gw0] [ 57%] PASSED test_storage_hdfs/test.py::test_read_write_storage_with_globs 
test_storage_hdfs/test.py::test_read_write_table 
[gw0] [ 58%] PASSED test_storage_hdfs/test.py::test_read_write_table 
test_storage_hdfs/test.py::test_read_write_table_with_parameter_none 
[gw0] [ 60%] PASSED test_storage_hdfs/test.py::test_read_write_table_with_parameter_none 
test_storage_hdfs/test.py::test_schema_inference 
[gw0] [ 61%] PASSED test_storage_hdfs/test.py::test_schema_inference 
test_storage_hdfs/test.py::test_schema_inference_cache 
[gw7] [ 62%] ERROR test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections 
test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials 
[gw7] [ 63%] ERROR test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials 
test_zookeeper_config/test_password.py::test_identity 
[gw0] [ 64%] PASSED test_storage_hdfs/test.py::test_schema_inference_cache 
test_storage_hdfs/test.py::test_schema_inference_with_globs 
[gw0] [ 65%] PASSED test_storage_hdfs/test.py::test_schema_inference_with_globs 
test_storage_hdfs/test.py::test_seekable_formats 
[gw3] [ 66%] PASSED test_storage_kerberized_hdfs/test.py::test_read_table_expired 
test_storage_kerberized_hdfs/test.py::test_read_write_storage 
[gw0] [ 67%] PASSED test_storage_hdfs/test.py::test_seekable_formats 
test_storage_hdfs/test.py::test_skip_empty_files 
[gw0] [ 68%] PASSED test_storage_hdfs/test.py::test_skip_empty_files 
test_storage_hdfs/test.py::test_storage_with_multidirectory_glob 
[gw0] [ 70%] PASSED test_storage_hdfs/test.py::test_storage_with_multidirectory_glob 
test_storage_hdfs/test.py::test_truncate_table 
[gw0] [ 71%] PASSED test_storage_hdfs/test.py::test_truncate_table 
test_storage_hdfs/test.py::test_virtual_columns 
[gw0] [ 72%] PASSED test_storage_hdfs/test.py::test_virtual_columns 
test_storage_hdfs/test.py::test_virtual_columns_2 
[gw3] [ 73%] PASSED test_storage_kerberized_hdfs/test.py::test_read_write_storage 
test_storage_kerberized_hdfs/test.py::test_two_users 
[gw0] [ 74%] PASSED test_storage_hdfs/test.py::test_virtual_columns_2 
test_storage_hdfs/test.py::test_write_gz_storage 
[gw7] [ 75%] PASSED test_zookeeper_config/test_password.py::test_identity 
[gw0] [ 76%] PASSED test_storage_hdfs/test.py::test_write_gz_storage 
test_storage_hdfs/test.py::test_write_gzip_storage 
[gw3] [ 77%] PASSED test_storage_kerberized_hdfs/test.py::test_two_users 
test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired 
[gw0] [ 78%] PASSED test_storage_hdfs/test.py::test_write_gzip_storage 
test_storage_hdfs/test.py::test_write_table 
[gw0] [ 80%] PASSED test_storage_hdfs/test.py::test_write_table 
[gw3] [ 81%] PASSED test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired 
[gw1] [ 82%] PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_cache 
test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs 
[gw1] [ 83%] PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs 
test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf 
[gw1] [ 84%] PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf 
test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs 
[gw1] [ 85%] PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs 
test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf 
[gw1] [ 86%] PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf 
test_storage_azure_blob_storage/test.py::test_simple_read_write 
[gw1] [ 87%] PASSED test_storage_azure_blob_storage/test.py::test_simple_read_write 
test_storage_azure_blob_storage/test.py::test_simple_write_account_string 
[gw1] [ 88%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_account_string 
test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function 
[gw1] [ 90%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function 
test_storage_azure_blob_storage/test.py::test_simple_write_connection_string 
[gw1] [ 91%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_connection_string 
test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function 
[gw1] [ 92%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function 
test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1 
[gw1] [ 93%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1 
test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function 
[gw1] [ 94%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function 
test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2 
[gw1] [ 95%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2 
test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function 
[gw1] [ 96%] PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function 
test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin] 
[gw1] [ 97%] PASSED test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin] 
test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz] 
[gw1] [ 98%] PASSED test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz] 
test_storage_azure_blob_storage/test.py::test_truncate 
[gw1] [100%] PASSED test_storage_azure_blob_storage/test.py::test_truncate 

==================================== ERRORS ====================================
_______________ ERROR at setup of test_no_sign_named_collections _______________
[gw7] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "s3_with_invalid_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "aws",
                    "AWS_SECRET_ACCESS_KEY": "aws123",
                },
                main_configs=[
                    "configs/use_environment_credentials.xml",
                    "configs/named_collections.xml",
                ],
                user_configs=["configs/users.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test_invalid_env_credentials.py:104: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test_invalid_env_credentials.py:68: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fd6cb3c7eb0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
---------------------------- Captured stdout setup -----------------------------
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
---------------------------- Captured stderr setup -----------------------------
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
------------------------------ Captured log setup ------------------------------
2026-10-27 17:29:37 [ 378 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:29:37 [ 378 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check)
2026-10-27 17:29:37 [ 378 ] DEBUG : No running containers (conftest.py:60, cleanup_environment)
2026-10-27 17:29:37 [ 378 ] DEBUG : Pruning Docker networks (conftest.py:62, cleanup_environment)
2026-10-27 17:29:37 [ 378 ] DEBUG : Command:['docker network prune --force'] (cluster.py:105, run_and_check)
2026-10-27 17:29:37 [ 378 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:105, run_and_check)
2026-10-27 17:29:37 [ 378 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:113, run_and_check)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                  DOCKER_KERBEROS_KDC_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV         CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                             MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                        JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                             TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                 HOSTNAME acf8b56b85a3 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                    SHLVL 0 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                     HOME /root (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                   OLDPWD / (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                        DOCKER_HELPER_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                         PYTHONUNBUFFERED 1 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                            UBSAN_OPTIONS print_stacktrace=1 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                           PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_storage_azure_blob_storage/test.py::test_partition_by_const_column test_storage_azure_blob_storage/test.py::test_partition_by_string_column test_storage_azure_blob_storage/test.py::test_partition_by_tf test_storage_azure_blob_storage/test.py::test_put_get_with_globs test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container test_storage_azure_blob_storage/test.py::test_read_subcolumns test_storage_azure_blob_storage/test.py::test_schema_inference_cache test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf test_storage_azure_blob_storage/test.py::test_simple_read_write test_storage_azure_blob_storage/test.py::test_simple_write_account_string test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function test_storage_azure_blob_storage/test.py::test_simple_write_connection_string test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1 test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2 test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function 'test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]' 'test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]' test_storage_azure_blob_storage/test.py::test_truncate test_storage_hdfs/test.py::test_bad_hdfs_uri test_storage_hdfs/test.py::test_cluster_join test_storage_hdfs/test.py::test_cluster_macro test_storage_hdfs/test.py::test_format_detection test_storage_hdfs/test.py::test_globs_in_read_table test_storage_hdfs/test.py::test_hdfsCluster test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards test_storage_hdfs/test.py::test_hdfs_directory_not_exist test_storage_hdfs/test.py::test_insert_select_schema_inference test_storage_hdfs/test.py::test_multiple_inserts test_storage_hdfs/test.py::test_overwrite test_storage_hdfs/test.py::test_partition_by test_storage_hdfs/test.py::test_read_files_with_spaces test_storage_hdfs/test.py::test_read_subcolumns test_storage_hdfs/test.py::test_read_table_with_default test_storage_hdfs/test.py::test_read_write_gzip_table test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip test_storage_hdfs/test.py::test_read_write_storage test_storage_hdfs/test.py::test_read_write_storage_with_globs test_storage_hdfs/test.py::test_read_write_table test_storage_hdfs/test.py::test_read_write_table_with_parameter_none test_storage_hdfs/test.py::test_schema_inference test_storage_hdfs/test.py::test_schema_inference_cache test_storage_hdfs/test.py::test_schema_inference_with_globs test_storage_hdfs/test.py::test_seekable_formats test_storage_hdfs/test.py::test_skip_empty_files test_storage_hdfs/test.py::test_storage_with_multidirectory_glob test_storage_hdfs/test.py::test_truncate_table test_storage_hdfs/test.py::test_virtual_columns test_storage_hdfs/test.py::test_virtual_columns_2 test_storage_hdfs/test.py::test_write_gz_storage test_storage_hdfs/test.py::test_write_gzip_storage test_storage_hdfs/test.py::test_write_table test_storage_kerberized_hdfs/test.py::test_cache_path test_storage_kerberized_hdfs/test.py::test_prohibited test_storage_kerberized_hdfs/test.py::test_read_table test_storage_kerberized_hdfs/test.py::test_read_table_expired test_storage_kerberized_hdfs/test.py::test_read_write_storage test_storage_kerberized_hdfs/test.py::test_two_users test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials 'test_storage_url/test.py::test_file_formats[CSV]' 'test_storage_url/test.py::test_file_formats[JSONEachRow]' 'test_storage_url/test.py::test_file_formats[Parquet]' 'test_storage_url/test.py::test_file_formats[TSV]' test_storage_url/test.py::test_partition_by test_storage_url/test.py::test_table_function_url_access_rights test_storage_url/test.py::test_url_cluster test_storage_url/test.py::test_url_cluster_with_named_collection test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric test_system_metrics/test.py::test_broken_tables_readonly_metric test_system_metrics/test.py::test_metrics_storage_buffer_size test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name test_wrong_db_or_table_name/test.py::test_wrong_database_name test_wrong_db_or_table_name/test.py::test_wrong_table_name test_zero_copy_fetch/test.py::test_concurrent_move_to_s3 test_zero_copy_fetch/test.py::test_fetch_correct_volume test_zero_copy_fetch/test.py::test_zero_copy_mutation test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part test_zookeeper_config/test_password.py::test_identity  -vvv (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV    CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                     COMPOSE_HTTP_TIMEOUT 600 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV              DOCKER_MYSQL_PHP_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                 DOCKER_DOTNET_CLIENT_TAG latest (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV         CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV               DOCKER_MYSQL_JS_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                     PATH /spark-3.3.2-bin-hadoop3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV             DOCKER_KERBERIZED_HADOOP_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                           DOCKER_CHANNEL stable (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                    DOCKER_CLIENT_TIMEOUT 300 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV        DOCKER_POSTGRESQL_JAVA_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                     DOCKER_NGINX_DAV_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV           DOCKER_MYSQL_GOLANG_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                      PWD /ClickHouse/tests/integration (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV             DOCKER_MYSQL_JAVA_CLIENT_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV       CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV         CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                       TZ Etc/UTC (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                          DOCKER_BASE_TAG 1864 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                               SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                                 LC_CTYPE C.UTF-8 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                 INTEGRATION_TESTS_RUN_ID 0 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                  PYTEST_XDIST_TESTRUNUID 79097cc3d4684091aa6ae9ecbc67901c (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                      PYTEST_XDIST_WORKER gw7 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : ENV                      PYTEST_CURRENT_TEST test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections (setup) (cluster.py:343, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:636, __init__)
2026-10-27 17:29:37 [ 378 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1629, add_instance)
2026-10-27 17:29:37 [ 378 ] DEBUG : Cluster name:invalid_env_credentials project_name:rootteststorages3invalidenvcredentials. Added instance name:s3_with_invalid_environment_credentials tag:1864 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', '--project-name', 'rootteststorages3invalidenvcredentials', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/compose (cluster.py:1915, add_instance)
2026-10-27 17:29:37 [ 378 ] INFO : Starting cluster... (test_invalid_env_credentials.py:98, started_cluster)
2026-10-27 17:29:37 [ 378 ] INFO : Running tests in /ClickHouse/tests/integration/test_storage_s3/test_invalid_env_credentials.py (cluster.py:2637, start)
2026-10-27 17:29:37 [ 378 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start)
2026-10-27 17:29:37 [ 378 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 35
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-130-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.59GiB
 Name: acf8b56b85a3
 ID: 38ee18d9-7524-4ca7-9a78-b5a339417f66
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Docker networks for project rootteststorages3invalidenvcredentials are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Docker containers for project rootteststorages3invalidenvcredentials are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Docker volumes for project rootteststorages3invalidenvcredentials are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 35
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-130-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.59GiB
 Name: acf8b56b85a3
 ID: 38ee18d9-7524-4ca7-9a78-b5a339417f66
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Docker networks for project rootteststorages3invalidenvcredentials are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Docker containers for project rootteststorages3invalidenvcredentials are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Docker volumes for project rootteststorages3invalidenvcredentials are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:29:38 [ 378 ] DEBUG : Command:docker container list --all --filter name='^/rootteststorages3invalidenvcredentials_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:29:38 [ 378 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : No running containers for project: rootteststorages3invalidenvcredentials (cluster.py:782, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:29:38 [ 378 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:29:38 [ 378 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:29:38 [ 378 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:29:38 [ 378 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check)
2026-10-27 17:29:38 [ 378 ] DEBUG : Setup directory for instance: s3_with_invalid_environment_credentials (cluster.py:2657, start)
2026-10-27 17:29:38 [ 378 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_s3/configs/use_environment_credentials.xml', '/ClickHouse/tests/integration/test_storage_s3/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/configs/config.d (cluster.py:4383, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/database (cluster.py:4400, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/logs (cluster.py:4411, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir)
2026-10-27 17:29:38 [ 378 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'AWS_ACCESS_KEY_ID': 'aws', 'AWS_SECRET_ACCESS_KEY': 'aws123', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/minio/certs/public.crt'} stored in /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env (cluster.py:78, _create_env_file)
2026-10-27 17:29:38 [ 378 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:29:38 [ 378 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:29:38 [ 378 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-10-27 17:29:38 [ 378 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-10-27 17:29:38 [ 378 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 825 (connectionpool.py:546, _make_request)
2026-10-27 17:29:38 [ 378 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', '--project-name', 'rootteststorages3invalidenvcredentials', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', 'pull'] (cluster.py:105, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy2                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling minio1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling s3_with_invalid_environment_credentials ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling resolver                                ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy2                                  ... pulling from altinityinfra/s3-proxy (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling resolver                                ... pulling from altinityinfra/python... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling minio1                                  ... pulling from clickhouse/minio-minio (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy2                                  ... digest: sha256:1a92e43060168b5557... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy2                                  ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling s3_with_invalid_environment_credentials ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy1                                  ... pulling from altinityinfra/s3-proxy (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy2                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling resolver                                ... digest: sha256:5a7a0ebe149009e702... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling resolver                                ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling resolver                                ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy1                                  ... digest: sha256:1a92e43060168b5557... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy1                                  ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling proxy1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling minio1                                  ... digest: sha256:9eea83a4e1425067e7... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling minio1                                  ... status: image is up to date for c... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling s3_with_invalid_environment_credentials ... digest: sha256:45997363e17ff1831d... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling s3_with_invalid_environment_credentials ... status: image is up to date for a... (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling minio1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] DEBUG : Stderr:Pulling s3_with_invalid_environment_credentials ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:49 [ 378 ] INFO : Trying to create Minio instance by command docker-compose --env-file /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env --project-name rootteststorages3invalidenvcredentials --file /compose/docker_compose_minio.yml --verbose up -d (cluster.py:2938, start)
2026-10-27 17:29:49 [ 378 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', '--project-name', 'rootteststorages3invalidenvcredentials', '--file', '/compose/docker_compose_minio.yml', '--verbose', 'up', '-d'] (cluster.py:105, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /compose/docker_compose_minio.yml (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:docker-py version: <module 'docker.version' from '/usr/local/lib/python3.10/dist-packages/docker/version.py'> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-130-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': 'v2.4.2', 'Details': {'GitCommit': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}}, {'Name': 'runc', 'Version': '1.5.2', 'Details': {'GitCommit': 'v1.5.2-0-g29dd3dc2'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-130-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststorages3invalidenvcredentials_default') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('rootteststorages3invalidenvcredentials_data1-1') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'b7a25261e029d68978bbf290df0a62fc819b3c51', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:                      'ID': 'b7a25261e029d68978bbf290df0a62fc819b3c51'}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Containers': 0, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststorages3invalidenvcredentials_default') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.network.ensure: Creating network "rootteststorages3invalidenvcredentials_default" with the default driver (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='rootteststorages3invalidenvcredentials_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': 'e5a5ecb8967fea44ad3bb65e51d1cd1113b6efad8d24a9936873460aad203ded', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('rootteststorages3invalidenvcredentials_data1-1') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.volume.initialize: Creating volume "rootteststorages3invalidenvcredentials_data1-1" with default driver (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume <- ('rootteststorages3invalidenvcredentials_data1-1', None, None, labels={'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', 'com.docker.compose.volume': 'data1-1', 'com.docker.compose.version': '1.29.2'}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume -> {'CreatedAt': '2026-10-27T17:29:49Z', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Driver': 'local', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Labels': {'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'com.docker.compose.version': '1.29.2', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'com.docker.compose.volume': 'data1-1'}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Mountpoint': '/var/lib/docker/volumes/rootteststorages3invalidenvcredentials_data1-1/_data', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Name': 'rootteststorages3invalidenvcredentials_data1-1', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Options': None, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Scope': 'local'} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['minio'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.project._get_convergence_plans: minio1 has upstream changes (proxy1, proxy2) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.project._get_convergence_plans: resolver has upstream changes (proxy1, proxy2) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: proxy1>, <Service: proxy2>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: proxy1> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: proxy2> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_proxy2_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_proxy1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3invalidenvcredentials', service='proxy2', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3invalidenvcredentials', service='proxy2', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3invalidenvcredentials', service='proxy1', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3invalidenvcredentials', service='proxy1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 7ca82a23d88e1568ab9149ff86a2b71757f09a70d3dfbf10ae6193191335ee66 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststorages3invalidenvcredentials_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3invalidenvcredentials_default', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='rootteststorages3invalidenvcredentials_proxy1_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', 'com.docker.compose.service': 'proxy1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '7ca82a23d88e1568ab9149ff86a2b71757f09a70d3dfbf10ae6193191335ee66'}, host_config={'NetworkMode': 'rootteststorages3invalidenvcredentials_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3invalidenvcredentials_default': {'Aliases': ['proxy1'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 7ca82a23d88e1568ab9149ff86a2b71757f09a70d3dfbf10ae6193191335ee66 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststorages3invalidenvcredentials_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3invalidenvcredentials_default', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='rootteststorages3invalidenvcredentials_proxy2_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', 'com.docker.compose.service': 'proxy2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '7ca82a23d88e1568ab9149ff86a2b71757f09a70d3dfbf10ae6193191335ee66'}, host_config={'NetworkMode': 'rootteststorages3invalidenvcredentials_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3invalidenvcredentials_default': {'Aliases': ['proxy2'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'd6c517775c59f96935c753a26e5cb15c6b9c926db73c175e1dc5828dbda858a6', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Warnings': ["The requested image's platform (linux/arm64/v8) does not match " (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:              'the detected host platform (linux/amd64/v4) and no specific ' (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:              'platform was requested']} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('d6c517775c59f96935c753a26e5cb15c6b9c926db73c175e1dc5828dbda858a6') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'a966bc3be4dd7f8b87d325b720e1f132f6ce29846635010cd8756954f5531e0f', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Warnings': ["The requested image's platform (linux/arm64/v8) does not match " (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:              'the detected host platform (linux/amd64/v4) and no specific ' (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:              'platform was requested']} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('a966bc3be4dd7f8b87d325b720e1f132f6ce29846635010cd8756954f5531e0f') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:                    'NGINX_VERSION=1.31.8', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('a966bc3be4dd7f8b87d325b720e1f132f6ce29846635010cd8756954f5531e0f', 'rootteststorages3invalidenvcredentials_default') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['/run.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:                    'NGINX_VERSION=1.31.8', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('d6c517775c59f96935c753a26e5cb15c6b9c926db73c175e1dc5828dbda858a6', 'rootteststorages3invalidenvcredentials_default') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('a966bc3be4dd7f8b87d325b720e1f132f6ce29846635010cd8756954f5531e0f', 'rootteststorages3invalidenvcredentials_default', aliases=['proxy1', 'a966bc3be4dd'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('d6c517775c59f96935c753a26e5cb15c6b9c926db73c175e1dc5828dbda858a6', 'rootteststorages3invalidenvcredentials_default', aliases=['proxy2', 'd6c517775c59'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('d6c517775c59f96935c753a26e5cb15c6b9c926db73c175e1dc5828dbda858a6') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('a966bc3be4dd7f8b87d325b720e1f132f6ce29846635010cd8756954f5531e0f') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3invalidenvcredentials', service='proxy1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_proxy1_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: proxy1> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3invalidenvcredentials', service='proxy2', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_proxy2_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: proxy2> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {<Service: minio1>, <Service: resolver>} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: minio1> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for <Service: resolver> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststorages3invalidenvcredentials', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_minio1_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3invalidenvcredentials', service='minio1', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3invalidenvcredentials', service='minio1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_resolver_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststorages3invalidenvcredentials', service='resolver', number=1)} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststorages3invalidenvcredentials', service='resolver', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['minio'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['minio'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 1b66731c4958cd79ff407437d37a9bf4af87f9a5055e65259fbd4a721d428e53 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'arm64', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Author': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=['/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/minio/certs:/certs:rw', 'rootteststorages3invalidenvcredentials_data1-1:/data1:rw'], volumes_from=[], privileged=False, network_mode='rootteststorages3invalidenvcredentials_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: a95a9180e47aa8a92c9755f9efc5553b985d0ff8a2a37378ecf4a1222eb11f15 (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': ['/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/minio/certs:/certs:rw', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:           'rootteststorages3invalidenvcredentials_data1-1:/data1:rw'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3invalidenvcredentials_default', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststorages3invalidenvcredentials_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='server --console-address 127.0.0.1:19001 --address :9001 --certs-dir /certs /data1-1', environment=['MINIO_ACCESS_KEY=minio', 'MINIO_SECRET_KEY=minio123', 'MINIO_PROMETHEUS_AUTH_TYPE=public'], image='clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z', volumes={'/certs': {}, '/data1': {}}, name='rootteststorages3invalidenvcredentials_minio1_1', detach=True, ports=['9001'], labels={'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', 'com.docker.compose.service': 'minio1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '1b66731c4958cd79ff407437d37a9bf4af87f9a5055e65259fbd4a721d428e53'}, host_config={'NetworkMode': 'rootteststorages3invalidenvcredentials_default', 'VolumesFrom': [], 'Binds': ['/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/minio/certs:/certs:rw', 'rootteststorages3invalidenvcredentials_data1-1:/data1:rw'], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3invalidenvcredentials_default': {'Aliases': ['minio1'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Links': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'NetworkMode': 'rootteststorages3invalidenvcredentials_default', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/python-bottle', tty=True, volumes={}, name='rootteststorages3invalidenvcredentials_resolver_1', detach=True, ports=['8080'], environment=[], labels={'com.docker.compose.project': 'rootteststorages3invalidenvcredentials', 'com.docker.compose.service': 'resolver', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/compose', 'com.docker.compose.project.config_files': '/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'a95a9180e47aa8a92c9755f9efc5553b985d0ff8a2a37378ecf4a1222eb11f15'}, host_config={'NetworkMode': 'rootteststorages3invalidenvcredentials_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststorages3invalidenvcredentials_default': {'Aliases': ['resolver'], 'IPAMConfig': {}}}}) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '29147e69b3130802ae061427f5cfff1f7ebf8c4907fc03427f04e67d7f8b86ab', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Warnings': ["The requested image's platform (linux/arm64/v8) does not match " (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:              'the detected host platform (linux/amd64/v4) and no specific ' (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:              'platform was requested']} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('29147e69b3130802ae061427f5cfff1f7ebf8c4907fc03427f04e67d7f8b86ab') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Args': [], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdin': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'AttachStdout': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Cmd': ['python3'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Domainname': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Entrypoint': None, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:            'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:                    'LANG=C.UTF-8', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('29147e69b3130802ae061427f5cfff1f7ebf8c4907fc03427f04e67d7f8b86ab', 'rootteststorages3invalidenvcredentials_default') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '7f127165edcb2805046066707a35a447dd26d3916533f84a87baa9e057cffc9f', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('7f127165edcb2805046066707a35a447dd26d3916533f84a87baa9e057cffc9f') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Args': ['server', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          '--console-address', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          '127.0.0.1:19001', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          '--address', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          ':9001', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          '--certs-dir', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          '/certs', (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:          '/data1-1'], (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:... (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('7f127165edcb2805046066707a35a447dd26d3916533f84a87baa9e057cffc9f', 'rootteststorages3invalidenvcredentials_default') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('29147e69b3130802ae061427f5cfff1f7ebf8c4907fc03427f04e67d7f8b86ab', 'rootteststorages3invalidenvcredentials_default', aliases=['resolver', '29147e69b313'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('7f127165edcb2805046066707a35a447dd26d3916533f84a87baa9e057cffc9f', 'rootteststorages3invalidenvcredentials_default', aliases=['7f127165edcb', 'minio1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('29147e69b3130802ae061427f5cfff1f7ebf8c4907fc03427f04e67d7f8b86ab') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('7f127165edcb2805046066707a35a447dd26d3916533f84a87baa9e057cffc9f') (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3invalidenvcredentials', service='resolver', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_resolver_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: resolver> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststorages3invalidenvcredentials', service='minio1', number=1) (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_minio1_1   ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: <Service: minio1> (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:115, run_and_check)
2026-10-27 17:29:51 [ 378 ] INFO : Trying to connect to Minio... (cluster.py:2944, start)
2026-10-27 17:29:51 [ 378 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:1971, get_instance_ip)
2026-10-27 17:29:51 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3invalidenvcredentials_minio1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:51 [ 378 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:1971, get_instance_ip)
2026-10-27 17:29:51 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3invalidenvcredentials_proxy1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:51 [ 378 ] DEBUG : Starting new HTTP connection (1): 172.16.3.5:9001 (connectionpool.py:244, _new_conn)
2026-10-27 17:29:51 [ 378 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment)
2026-10-27 17:29:51 [ 378 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6cb2cec20>: Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:871, urlopen)
2026-10-27 17:29:51 [ 378 ] DEBUG : Starting new HTTP connection (2): 172.16.3.5:9001 (connectionpool.py:244, _new_conn)
2026-10-27 17:29:51 [ 378 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment)
2026-10-27 17:29:51 [ 378 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6cb2cee60>: Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:871, urlopen)
2026-10-27 17:29:51 [ 378 ] DEBUG : Starting new HTTP connection (3): 172.16.3.5:9001 (connectionpool.py:244, _new_conn)
2026-10-27 17:29:51 [ 378 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment)
2026-10-27 17:29:51 [ 378 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6cb2cf0a0>: Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:871, urlopen)
2026-10-27 17:29:51 [ 378 ] DEBUG : Starting new HTTP connection (4): 172.16.3.5:9001 (connectionpool.py:244, _new_conn)
2026-10-27 17:29:51 [ 378 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.3.5', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6cb2cf2e0>: Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2521, wait_minio_to_start)
2026-10-27 17:29:52 [ 378 ] DEBUG : Starting new HTTP connection (5): 172.16.3.5:9001 (connectionpool.py:244, _new_conn)
2026-10-27 17:29:52 [ 378 ] DEBUG : http://172.16.3.5:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:52 [ 378 ] DEBUG : Connected to Minio. (cluster.py:2501, wait_minio_to_start)
2026-10-27 17:29:52 [ 378 ] DEBUG : http://172.16.3.5:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:52 [ 378 ] DEBUG : http://172.16.3.5:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:52 [ 378 ] DEBUG : S3 bucket 'root' created (cluster.py:2516, wait_minio_to_start)
2026-10-27 17:29:52 [ 378 ] DEBUG : http://172.16.3.5:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:52 [ 378 ] DEBUG : http://172.16.3.5:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:52 [ 378 ] DEBUG : S3 bucket 'root2' created (cluster.py:2516, wait_minio_to_start)
2026-10-27 17:29:52 [ 378 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env --project-name rootteststorages3invalidenvcredentials --file /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/docker-compose.yml --file /compose/docker_compose_minio.yml up -d --no-recreate') (cluster.py:2980, start)
2026-10-27 17:29:52 [ 378 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', '--project-name', 'rootteststorages3invalidenvcredentials', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check)
2026-10-27 17:29:55 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:29:55 [ 378 ] DEBUG : Stderr:Creating rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:29:55 [ 378 ] DEBUG : ClickHouse instance created (cluster.py:2988, start)
2026-10-27 17:29:55 [ 378 ] DEBUG : get_instance_ip instance_name=s3_with_invalid_environment_credentials (cluster.py:1971, get_instance_ip)
2026-10-27 17:29:55 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:55 [ 378 ] DEBUG : Waiting for ClickHouse start in s3_with_invalid_environment_credentials, ip: 172.16.3.6... (cluster.py:2995, start)
2026-10-27 17:29:55 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:55 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/9c061a7228f73e9478f97efc3eff8378dcc86fcc7b786c22aeea913d8044c724/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:55 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/9c061a7228f73e9478f97efc3eff8378dcc86fcc7b786c22aeea913d8044c724/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:55 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/9c061a7228f73e9478f97efc3eff8378dcc86fcc7b786c22aeea913d8044c724/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:55 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/9c061a7228f73e9478f97efc3eff8378dcc86fcc7b786c22aeea913d8044c724/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:56 [ 378 ] DEBUG : http://localhost:None "GET /v1.42/containers/9c061a7228f73e9478f97efc3eff8378dcc86fcc7b786c22aeea913d8044c724/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request)
2026-10-27 17:29:56 [ 378 ] DEBUG : ClickHouse s3_with_invalid_environment_credentials started (cluster.py:2999, start)
2026-10-27 17:29:56 [ 378 ] INFO : Cluster started (test_invalid_env_credentials.py:100, started_cluster)
2026-10-27 17:29:56 [ 378 ] DEBUG : http://172.16.3.5:9001 "PUT /root?policy= HTTP/1.1" 204 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:56 [ 378 ] DEBUG : http://172.16.3.5:9001 "GET /root-with-auth?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:56 [ 378 ] DEBUG : http://172.16.3.5:9001 "PUT /root-with-auth HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2026-10-27 17:29:56 [ 378 ] INFO : S3 bucket created (test_invalid_env_credentials.py:103, started_cluster)
2026-10-27 17:29:56 [ 378 ] INFO : Starting mock servers mock_s3.py,unstable_server.py,echo.py,no_list_objects.py (mock_servers.py:18, start_mock_servers)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJvdXRlLCBydW4sIHJlcXVlc3QsIHJlc3BvbnNlCgoKQHJvdXRlKCIvcmVkaXJlY3RlZC88X3BhdGg6cGF0aD4iKQpkZWYgaW5maW5pdGVfcmVkaXJlY3QoX3BhdGgpOgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiTG9jYXRpb24iLCByZXF1ZXN0LnVybCkKICAgIHJlc3BvbnNlLnN0YXR1cyA9IDMwNwogICAgcmV0dXJuICJSZWRpcmVjdGVkIgoKCkByb3V0ZSgiLzxfYnVja2V0Pi88X3BhdGg6cGF0aD4iKQpkZWYgc2VydmVyKF9idWNrZXQsIF9wYXRoKToKICAgIGZvciBuYW1lIGluIHJlcXVlc3QuaGVhZGVyczoKICAgICAgICBpZiBuYW1lID09ICJBdXRob3JpemF0aW9uIiBhbmQgcmVxdWVzdC5oZWFkZXJzW25hbWVdID09ICJCZWFyZXIgVE9LRU4iOgogICAgICAgICAgICByZXN1bHQgPSAiMSwgMiwgMyIKICAgICAgICAgICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICAgICAgICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHJlc3VsdCkpCiAgICAgICAgICAgIHJldHVybiByZXN1bHQKCiAgICByZXN1bHQgPSAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48RXJyb3I+PENvZGU+Rm9yYmlkZGVuRXJyb3I8L0NvZGU+PE1lc3NhZ2U+Rm9yYmlkZGVuIEVycm9yPC9NZXNzYWdlPjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPjwvRXJyb3I+JwogICAgcmVzcG9uc2Uuc3RhdHVzID0gNDAzCiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC94bWwiCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihyZXN1bHQpKQogICAgcmV0dXJuIHJlc3VsdAoKCkByb3V0ZSgiLyIpCmRlZiBwaW5nKCk6CiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC9wbGFpbiIKICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMikKICAgIHJldHVybiAiT0siCgoKcnVuKGhvc3Q9IjAuMC4wLjAiLCBwb3J0PWludChzeXMuYXJndlsxXSkpCg== | base64 --decode > mock_s3.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJvdXRlLCBydW4sIHJlcXVlc3QsIHJlc3BvbnNlCgoKQHJvdXRlKCIvcmVkaXJlY3RlZC88X3BhdGg6cGF0aD4iKQpkZWYgaW5maW5pdGVfcmVkaXJlY3QoX3BhdGgpOgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiTG9jYXRpb24iLCByZXF1ZXN0LnVybCkKICAgIHJlc3BvbnNlLnN0YXR1cyA9IDMwNwogICAgcmV0dXJuICJSZWRpcmVjdGVkIgoKCkByb3V0ZSgiLzxfYnVja2V0Pi88X3BhdGg6cGF0aD4iKQpkZWYgc2VydmVyKF9idWNrZXQsIF9wYXRoKToKICAgIGZvciBuYW1lIGluIHJlcXVlc3QuaGVhZGVyczoKICAgICAgICBpZiBuYW1lID09ICJBdXRob3JpemF0aW9uIiBhbmQgcmVxdWVzdC5oZWFkZXJzW25hbWVdID09ICJCZWFyZXIgVE9LRU4iOgogICAgICAgICAgICByZXN1bHQgPSAiMSwgMiwgMyIKICAgICAgICAgICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICAgICAgICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHJlc3VsdCkpCiAgICAgICAgICAgIHJldHVybiByZXN1bHQKCiAgICByZXN1bHQgPSAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48RXJyb3I+PENvZGU+Rm9yYmlkZGVuRXJyb3I8L0NvZGU+PE1lc3NhZ2U+Rm9yYmlkZGVuIEVycm9yPC9NZXNzYWdlPjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPjwvRXJyb3I+JwogICAgcmVzcG9uc2Uuc3RhdHVzID0gNDAzCiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC94bWwiCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihyZXN1bHQpKQogICAgcmV0dXJuIHJlc3VsdAoKCkByb3V0ZSgiLyIpCmRlZiBwaW5nKCk6CiAgICByZXNwb25zZS5jb250ZW50X3R5cGUgPSAidGV4dC9wbGFpbiIKICAgIHJlc3BvbnNlLnNldF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMikKICAgIHJldHVybiAiT0siCgoKcnVuKGhvc3Q9IjAuMC4wLjAiLCBwb3J0PWludChzeXMuYXJndlsxXSkpCg== | base64 --decode > mock_s3.py'] (cluster.py:105, run_and_check)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:True nothrow:False cmd: ['python', 'mock_s3.py', '8080'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCByYW5kb20KaW1wb3J0IHJlCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAppbXBvcnQgc3lzCgoKZGVmIGdlbl9uX2RpZ2l0X251bWJlcihuKToKICAgIGFzc2VydCAwIDwgbiA8IDE5CiAgICByZXR1cm4gcmFuZG9tLnJhbmRpbnQoMTAgKiogKG4gLSAxKSwgMTAqKm4gLSAxKQoKCnN1bV9pbl80X2NvbHVtbiA9IDAKCgpkZWYgZ2VuX2xpbmUoKToKICAgIGdsb2JhbCBzdW1faW5fNF9jb2x1bW4KICAgIGNvbHVtbnMgPSA0CgogICAgcm93ID0gW10KCiAgICBkZWYgYWRkX251bWJlcigpOgogICAgICAgIGRpZ2l0cyA9IHJhbmRvbS5yYW5kaW50KDEsIDE4KQogICAgICAgIHJvdy5hcHBlbmQoZ2VuX25fZGlnaXRfbnVtYmVyKGRpZ2l0cykpCgogICAgZm9yIGkgaW4gcmFuZ2UoY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHJvdy5hcHBlbmQoMSkKICAgIGZvciBpIGluIHJhbmdlKGNvbHVtbnMgLSAxIC0gY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHN1bV9pbl80X2NvbHVtbiArPSByb3dbLTFdCgogICAgbGluZSA9ICIsIi5qb2luKG1hcChzdHIsIHJvdykpICsgIlxuIgogICAgcmV0dXJuIGxpbmUuZW5jb2RlKCkKCgpyYW5kb20uc2VlZCgiVW5zdGFibGUgc2VydmVyLzEuMCIpCgojIEdlbmVyYXRpbmcgc29tZSAicmFuZG9tIiBkYXRhIGFuZCBhcHBlbmQgYSBsaW5lIHdoaWNoIGNvbnRhaW5zIHN1bSBvZiBudW1iZXJzIGluIGNvbHVtbiA0LgpsaW5lcyA9ICgKICAgIGIiIi5qb2luKChnZW5fbGluZSgpIGZvciBfIGluIHJhbmdlKDUwMDAwMCkpKQogICAgKyBmIjAsMCwwLHstc3VtX2luXzRfY29sdW1ufVxuIi5lbmNvZGUoKQopCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBpZiBzZWxmLnBhdGggPT0gIi9yb290L3Rlc3QuY3N2IjoKICAgICAgICAgICAgc2VsZi5mcm9tX2J5dGVzID0gMAogICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGxlbihsaW5lcykKICAgICAgICAgICAgc2VsZi5zaXplID0gc2VsZi5lbmRfYnl0ZXMKICAgICAgICAgICAgc2VsZi5zZW5kX2Jsb2NrX3NpemUgPSAyNTYKICAgICAgICAgICAgc2VsZi5zdG9wX2F0ID0gKAogICAgICAgICAgICAgICAgcmFuZG9tLnJhbmRpbnQoOTAwMDAwLCAxMzAwMDAwKSAvLyBzZWxmLnNlbmRfYmxvY2tfc2l6ZQogICAgICAgICAgICApICAjIEJsb2NrIHNpemUgaXMgMTAyNCoqMi4KCiAgICAgICAgICAgIGlmICJSYW5nZSIgaW4gc2VsZi5oZWFkZXJzOgogICAgICAgICAgICAgICAgY3IgPSBzZWxmLmhlYWRlcnNbIlJhbmdlIl0KICAgICAgICAgICAgICAgIHBhcnRzID0gcmUuc3BsaXQoIlsgLS89XSsiLCBjcikKICAgICAgICAgICAgICAgIGFzc2VydCBwYXJ0c1swXSA9PSAiYnl0ZXMiCiAgICAgICAgICAgICAgICBzZWxmLmZyb21fYnl0ZXMgPSBpbnQocGFydHNbMV0pCiAgICAgICAgICAgICAgICBpZiBwYXJ0c1syXToKICAgICAgICAgICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGludChwYXJ0c1syXSkgKyAxCiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjA2KQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigKICAgICAgICAgICAgICAgICAgICAiQ29udGVudC1SYW5nZSIsCiAgICAgICAgICAgICAgICAgICAgZiJieXRlcyB7c2VsZi5mcm9tX2J5dGVzfS17c2VsZi5lbmRfYnl0ZXMtMX0ve3NlbGYuc2l6ZX0iLAogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkFjY2VwdC1SYW5nZXMiLCAiYnl0ZXMiKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgZiJ7c2VsZi5lbmRfYnl0ZXMtc2VsZi5mcm9tX2J5dGVzfSIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgICAgICBlbGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSg0MDQpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgZGVmIGRvX0dFVChzZWxmKToKICAgICAgICBzZWxmLmRvX0hFQUQoKQogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiL3Jvb3QvdGVzdC5jc3YiOgogICAgICAgICAgICBmb3IgYywgaSBpbiBlbnVtZXJhdGUoCiAgICAgICAgICAgICAgICByYW5nZShzZWxmLmZyb21fYnl0ZXMsIHNlbGYuZW5kX2J5dGVzLCBzZWxmLnNlbmRfYmxvY2tfc2l6ZSkKICAgICAgICAgICAgKToKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgbGluZXNbaSA6IG1pbihpICsgc2VsZi5zZW5kX2Jsb2NrX3NpemUsIHNlbGYuZW5kX2J5dGVzKV0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIGlmIChjICsgMSkgJSBzZWxmLnN0b3BfYXQgPT0gMDoKICAgICAgICAgICAgICAgICAgICAjIHNlbGYud2ZpbGUuX3NvY2suc2V0c29ja29wdChzb2NrZXQuU09MX1NPQ0tFVCwgc29ja2V0LlNPX0xJTkdFUiwgc3RydWN0LnBhY2soImlpIiwgMCwgMCkpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLnNodXRkb3duKHNvY2tldC5TSFVUX1JEV1IpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLmNsb3NlKCkKICAgICAgICAgICAgICAgICAgICBwcmludCgiRHJvcHBpbmcgY29ubmVjdGlvbiIpCiAgICAgICAgICAgICAgICAgICAgYnJlYWsKCiAgICAgICAgZWxpZiBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLndmaWxlLndyaXRlKGIiT0siKQoKCmh0dHBkID0gaHR0cC5zZXJ2ZXIuSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCmh0dHBkLnNlcnZlX2ZvcmV2ZXIoKQo= | base64 --decode > unstable_server.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCByYW5kb20KaW1wb3J0IHJlCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAppbXBvcnQgc3lzCgoKZGVmIGdlbl9uX2RpZ2l0X251bWJlcihuKToKICAgIGFzc2VydCAwIDwgbiA8IDE5CiAgICByZXR1cm4gcmFuZG9tLnJhbmRpbnQoMTAgKiogKG4gLSAxKSwgMTAqKm4gLSAxKQoKCnN1bV9pbl80X2NvbHVtbiA9IDAKCgpkZWYgZ2VuX2xpbmUoKToKICAgIGdsb2JhbCBzdW1faW5fNF9jb2x1bW4KICAgIGNvbHVtbnMgPSA0CgogICAgcm93ID0gW10KCiAgICBkZWYgYWRkX251bWJlcigpOgogICAgICAgIGRpZ2l0cyA9IHJhbmRvbS5yYW5kaW50KDEsIDE4KQogICAgICAgIHJvdy5hcHBlbmQoZ2VuX25fZGlnaXRfbnVtYmVyKGRpZ2l0cykpCgogICAgZm9yIGkgaW4gcmFuZ2UoY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHJvdy5hcHBlbmQoMSkKICAgIGZvciBpIGluIHJhbmdlKGNvbHVtbnMgLSAxIC0gY29sdW1ucyAvLyAyKToKICAgICAgICBhZGRfbnVtYmVyKCkKICAgIHN1bV9pbl80X2NvbHVtbiArPSByb3dbLTFdCgogICAgbGluZSA9ICIsIi5qb2luKG1hcChzdHIsIHJvdykpICsgIlxuIgogICAgcmV0dXJuIGxpbmUuZW5jb2RlKCkKCgpyYW5kb20uc2VlZCgiVW5zdGFibGUgc2VydmVyLzEuMCIpCgojIEdlbmVyYXRpbmcgc29tZSAicmFuZG9tIiBkYXRhIGFuZCBhcHBlbmQgYSBsaW5lIHdoaWNoIGNvbnRhaW5zIHN1bSBvZiBudW1iZXJzIGluIGNvbHVtbiA0LgpsaW5lcyA9ICgKICAgIGIiIi5qb2luKChnZW5fbGluZSgpIGZvciBfIGluIHJhbmdlKDUwMDAwMCkpKQogICAgKyBmIjAsMCwwLHstc3VtX2luXzRfY29sdW1ufVxuIi5lbmNvZGUoKQopCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBpZiBzZWxmLnBhdGggPT0gIi9yb290L3Rlc3QuY3N2IjoKICAgICAgICAgICAgc2VsZi5mcm9tX2J5dGVzID0gMAogICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGxlbihsaW5lcykKICAgICAgICAgICAgc2VsZi5zaXplID0gc2VsZi5lbmRfYnl0ZXMKICAgICAgICAgICAgc2VsZi5zZW5kX2Jsb2NrX3NpemUgPSAyNTYKICAgICAgICAgICAgc2VsZi5zdG9wX2F0ID0gKAogICAgICAgICAgICAgICAgcmFuZG9tLnJhbmRpbnQoOTAwMDAwLCAxMzAwMDAwKSAvLyBzZWxmLnNlbmRfYmxvY2tfc2l6ZQogICAgICAgICAgICApICAjIEJsb2NrIHNpemUgaXMgMTAyNCoqMi4KCiAgICAgICAgICAgIGlmICJSYW5nZSIgaW4gc2VsZi5oZWFkZXJzOgogICAgICAgICAgICAgICAgY3IgPSBzZWxmLmhlYWRlcnNbIlJhbmdlIl0KICAgICAgICAgICAgICAgIHBhcnRzID0gcmUuc3BsaXQoIlsgLS89XSsiLCBjcikKICAgICAgICAgICAgICAgIGFzc2VydCBwYXJ0c1swXSA9PSAiYnl0ZXMiCiAgICAgICAgICAgICAgICBzZWxmLmZyb21fYnl0ZXMgPSBpbnQocGFydHNbMV0pCiAgICAgICAgICAgICAgICBpZiBwYXJ0c1syXToKICAgICAgICAgICAgICAgICAgICBzZWxmLmVuZF9ieXRlcyA9IGludChwYXJ0c1syXSkgKyAxCiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjA2KQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigKICAgICAgICAgICAgICAgICAgICAiQ29udGVudC1SYW5nZSIsCiAgICAgICAgICAgICAgICAgICAgZiJieXRlcyB7c2VsZi5mcm9tX2J5dGVzfS17c2VsZi5lbmRfYnl0ZXMtMX0ve3NlbGYuc2l6ZX0iLAogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkFjY2VwdC1SYW5nZXMiLCAiYnl0ZXMiKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgZiJ7c2VsZi5lbmRfYnl0ZXMtc2VsZi5mcm9tX2J5dGVzfSIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgICAgICBlbGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSg0MDQpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgZGVmIGRvX0dFVChzZWxmKToKICAgICAgICBzZWxmLmRvX0hFQUQoKQogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiL3Jvb3QvdGVzdC5jc3YiOgogICAgICAgICAgICBmb3IgYywgaSBpbiBlbnVtZXJhdGUoCiAgICAgICAgICAgICAgICByYW5nZShzZWxmLmZyb21fYnl0ZXMsIHNlbGYuZW5kX2J5dGVzLCBzZWxmLnNlbmRfYmxvY2tfc2l6ZSkKICAgICAgICAgICAgKToKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgbGluZXNbaSA6IG1pbihpICsgc2VsZi5zZW5kX2Jsb2NrX3NpemUsIHNlbGYuZW5kX2J5dGVzKV0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIGlmIChjICsgMSkgJSBzZWxmLnN0b3BfYXQgPT0gMDoKICAgICAgICAgICAgICAgICAgICAjIHNlbGYud2ZpbGUuX3NvY2suc2V0c29ja29wdChzb2NrZXQuU09MX1NPQ0tFVCwgc29ja2V0LlNPX0xJTkdFUiwgc3RydWN0LnBhY2soImlpIiwgMCwgMCkpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLnNodXRkb3duKHNvY2tldC5TSFVUX1JEV1IpCiAgICAgICAgICAgICAgICAgICAgIyBzZWxmLndmaWxlLl9zb2NrLmNsb3NlKCkKICAgICAgICAgICAgICAgICAgICBwcmludCgiRHJvcHBpbmcgY29ubmVjdGlvbiIpCiAgICAgICAgICAgICAgICAgICAgYnJlYWsKCiAgICAgICAgZWxpZiBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLndmaWxlLndyaXRlKGIiT0siKQoKCmh0dHBkID0gaHR0cC5zZXJ2ZXIuSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCmh0dHBkLnNlcnZlX2ZvcmV2ZXIoKQo= | base64 --decode > unstable_server.py'] (cluster.py:105, run_and_check)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:True nothrow:False cmd: ['python', 'unstable_server.py', '8081'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCBzeXMKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBnZXRfcmVzcG9uc2Uoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKToKICAgICAgICAgICAgcmV0dXJuIGIiLyIgKyBzZWxmLnBhdGguc3BsaXQoIi8iLCBtYXhzcGxpdD0yKVsyXS5lbmNvZGUoKQogICAgICAgIGVsaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgcmV0dXJuIGIiT0siCgogICAgICAgIHJldHVybiBOb25lCgogICAgZGVmIGRvX0hFQUQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKSBvciBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNDA0KQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBzZWxmLndmaWxlLndyaXRlKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpCgoKaHR0cGQgPSBodHRwLnNlcnZlci5IVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > echo.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCBzeXMKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBnZXRfcmVzcG9uc2Uoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKToKICAgICAgICAgICAgcmV0dXJuIGIiLyIgKyBzZWxmLnBhdGguc3BsaXQoIi8iLCBtYXhzcGxpdD0yKVsyXS5lbmNvZGUoKQogICAgICAgIGVsaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgcmV0dXJuIGIiT0siCgogICAgICAgIHJldHVybiBOb25lCgogICAgZGVmIGRvX0hFQUQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9nZXQtbXktcGF0aC8iKSBvciBzZWxmLnBhdGggPT0gIi8iOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgbGVuKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNDA0KQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAidGV4dC9wbGFpbiIpCiAgICAgICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBzZWxmLndmaWxlLndyaXRlKHNlbGYuZ2V0X3Jlc3BvbnNlKCkpCgoKaHR0cGQgPSBodHRwLnNlcnZlci5IVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > echo.py'] (cluster.py:105, run_and_check)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:True nothrow:False cmd: ['python', 'echo.py', '8082'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGh0dHAuY2xpZW50CmltcG9ydCBodHRwLnNlcnZlcgppbXBvcnQgcmFuZG9tCmltcG9ydCBzb2NrZXRzZXJ2ZXIKaW1wb3J0IHN5cwppbXBvcnQgdXJsbGliLnBhcnNlCgoKVVBTVFJFQU1fSE9TVCA9ICJtaW5pbzE6OTAwMSIKcmFuZG9tLnNlZWQoIk5vIGxpc3Qgb2JqZWN0cy8xLjAiKQoKbGlzdF9yZXF1ZXN0X2NvdW50ZXIgPSAwCmxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyID0gMTAKCgpkZWYgcmVxdWVzdChjb21tYW5kLCB1cmwsIGhlYWRlcnM9e30sIGRhdGE9Tm9uZSk6CiAgICAiIiJNaW5pLXJlcXVlc3RzLiIiIgoKICAgIGNsYXNzIER1bW15OgogICAgICAgIHBhc3MKCiAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxwYXJzZSh1cmwpCiAgICBjID0gaHR0cC5jbGllbnQuSFRUUENvbm5lY3Rpb24ocGFydHMuaG9zdG5hbWUsIHBhcnRzLnBvcnQpCiAgICBjLnJlcXVlc3QoCiAgICAgICAgY29tbWFuZCwKICAgICAgICB1cmxsaWIucGFyc2UudXJsdW5wYXJzZShwYXJ0cy5fcmVwbGFjZShzY2hlbWU9IiIsIG5ldGxvYz0iIikpLAogICAgICAgIGhlYWRlcnM9aGVhZGVycywKICAgICAgICBib2R5PWRhdGEsCiAgICApCiAgICByID0gYy5nZXRyZXNwb25zZSgpCiAgICByZXN1bHQgPSBEdW1teSgpCiAgICByZXN1bHQuc3RhdHVzX2NvZGUgPSByLnN0YXR1cwogICAgcmVzdWx0LmhlYWRlcnMgPSByLmhlYWRlcnMKICAgIHJlc3VsdC5jb250ZW50ID0gci5yZWFkKCkKICAgIHJldHVybiByZXN1bHQKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIk9LIikKICAgICAgICAgICAgcmV0dXJuCgogICAgICAgIHF1ZXJ5ID0gdXJsbGliLnBhcnNlLnVybHBhcnNlKHNlbGYucGF0aCkucXVlcnkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfY291bnRlcgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfbWF4X251bWJlcgoKICAgICAgICBpZiAibGlzdC10eXBlIiBpbiBwYXJhbXM6CiAgICAgICAgICAgIGlmIGxpc3RfcmVxdWVzdF9jb3VudGVyID4gbGlzdF9yZXF1ZXN0X21heF9udW1iZXI6CiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAxKQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgImFwcGxpY2F0aW9uL3htbCIpCiAgICAgICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgYiIiIjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+CiAgICAgICAgICAgICAgICAgICAgICAgIDxFcnJvcj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxDb2RlPk5vdEltcGxlbWVudGVkPC9Db2RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPE1lc3NhZ2U+SSBjYW4gbGlzdCBvYmplY3Qgb25seSBvbmNlPC9NZXNzYWdlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPFJlc291cmNlPlJFU09VUkNFPC9SZXNvdXJjZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxSZXF1ZXN0SWQ+UkVRVUVTVF9JRDwvUmVxdWVzdElkPgogICAgICAgICAgICAgICAgICAgICAgICA8L0Vycm9yPgogICAgICAgICAgICAgICAgICAgICIiIgogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X2NvdW50ZXIgKz0gMQogICAgICAgICAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19QVVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX1BPU1Qoc2VsZik6CiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9jb3VudGVyCiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyCgogICAgICAgIGlmIHNlbGYucGF0aC5zdGFydHN3aXRoKCIvcmVzZXRfY291bnRlcnMiKToKICAgICAgICAgICAgcXVlcnkgPSB1cmxsaWIucGFyc2UudXJscGFyc2Uoc2VsZi5wYXRoKS5xdWVyeQogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgICAgICBsaXN0X3JlcXVlc3RfY291bnRlciA9IDAKICAgICAgICAgICAgaWYgIm1heCIgaW4gcGFyYW1zOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X21heF9udW1iZXIgPSBpbnQocGFyYW1zWyJtYXgiXVswXSkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCiAgICAgICAgICAgIHJldHVybgoKICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19IRUFEKHNlbGYpOgogICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiKQogICAgICAgIGRhdGEgPSBzZWxmLnJmaWxlLnJlYWQoaW50KGNvbnRlbnRfbGVuZ3RoKSkgaWYgY29udGVudF9sZW5ndGggZWxzZSBOb25lCiAgICAgICAgciA9IHJlcXVlc3QoCiAgICAgICAgICAgIHNlbGYuY29tbWFuZCwKICAgICAgICAgICAgZiJodHRwOi8ve1VQU1RSRUFNX0hPU1R9e3NlbGYucGF0aH0iLAogICAgICAgICAgICBoZWFkZXJzPXNlbGYuaGVhZGVycywKICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2Uoci5zdGF0dXNfY29kZSkKICAgICAgICBmb3IgaywgdiBpbiByLmhlYWRlcnMuaXRlbXMoKToKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcihrLCB2KQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoci5jb250ZW50KQogICAgICAgIHNlbGYud2ZpbGUuY2xvc2UoKQoKCmNsYXNzIFRocmVhZGVkSFRUUFNlcnZlcihzb2NrZXRzZXJ2ZXIuVGhyZWFkaW5nTWl4SW4sIGh0dHAuc2VydmVyLkhUVFBTZXJ2ZXIpOgogICAgIiIiSGFuZGxlIHJlcXVlc3RzIGluIGEgc2VwYXJhdGUgdGhyZWFkLiIiIgoKCmh0dHBkID0gVGhyZWFkZWRIVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > no_list_objects.py'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGh0dHAuY2xpZW50CmltcG9ydCBodHRwLnNlcnZlcgppbXBvcnQgcmFuZG9tCmltcG9ydCBzb2NrZXRzZXJ2ZXIKaW1wb3J0IHN5cwppbXBvcnQgdXJsbGliLnBhcnNlCgoKVVBTVFJFQU1fSE9TVCA9ICJtaW5pbzE6OTAwMSIKcmFuZG9tLnNlZWQoIk5vIGxpc3Qgb2JqZWN0cy8xLjAiKQoKbGlzdF9yZXF1ZXN0X2NvdW50ZXIgPSAwCmxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyID0gMTAKCgpkZWYgcmVxdWVzdChjb21tYW5kLCB1cmwsIGhlYWRlcnM9e30sIGRhdGE9Tm9uZSk6CiAgICAiIiJNaW5pLXJlcXVlc3RzLiIiIgoKICAgIGNsYXNzIER1bW15OgogICAgICAgIHBhc3MKCiAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxwYXJzZSh1cmwpCiAgICBjID0gaHR0cC5jbGllbnQuSFRUUENvbm5lY3Rpb24ocGFydHMuaG9zdG5hbWUsIHBhcnRzLnBvcnQpCiAgICBjLnJlcXVlc3QoCiAgICAgICAgY29tbWFuZCwKICAgICAgICB1cmxsaWIucGFyc2UudXJsdW5wYXJzZShwYXJ0cy5fcmVwbGFjZShzY2hlbWU9IiIsIG5ldGxvYz0iIikpLAogICAgICAgIGhlYWRlcnM9aGVhZGVycywKICAgICAgICBib2R5PWRhdGEsCiAgICApCiAgICByID0gYy5nZXRyZXNwb25zZSgpCiAgICByZXN1bHQgPSBEdW1teSgpCiAgICByZXN1bHQuc3RhdHVzX2NvZGUgPSByLnN0YXR1cwogICAgcmVzdWx0LmhlYWRlcnMgPSByLmhlYWRlcnMKICAgIHJlc3VsdC5jb250ZW50ID0gci5yZWFkKCkKICAgIHJldHVybiByZXN1bHQKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIk9LIikKICAgICAgICAgICAgcmV0dXJuCgogICAgICAgIHF1ZXJ5ID0gdXJsbGliLnBhcnNlLnVybHBhcnNlKHNlbGYucGF0aCkucXVlcnkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfY291bnRlcgogICAgICAgIGdsb2JhbCBsaXN0X3JlcXVlc3RfbWF4X251bWJlcgoKICAgICAgICBpZiAibGlzdC10eXBlIiBpbiBwYXJhbXM6CiAgICAgICAgICAgIGlmIGxpc3RfcmVxdWVzdF9jb3VudGVyID4gbGlzdF9yZXF1ZXN0X21heF9udW1iZXI6CiAgICAgICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAxKQogICAgICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgImFwcGxpY2F0aW9uL3htbCIpCiAgICAgICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoCiAgICAgICAgICAgICAgICAgICAgYiIiIjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+CiAgICAgICAgICAgICAgICAgICAgICAgIDxFcnJvcj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxDb2RlPk5vdEltcGxlbWVudGVkPC9Db2RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPE1lc3NhZ2U+SSBjYW4gbGlzdCBvYmplY3Qgb25seSBvbmNlPC9NZXNzYWdlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPFJlc291cmNlPlJFU09VUkNFPC9SZXNvdXJjZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxSZXF1ZXN0SWQ+UkVRVUVTVF9JRDwvUmVxdWVzdElkPgogICAgICAgICAgICAgICAgICAgICAgICA8L0Vycm9yPgogICAgICAgICAgICAgICAgICAgICIiIgogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X2NvdW50ZXIgKz0gMQogICAgICAgICAgICAgICAgc2VsZi5kb19IRUFEKCkKICAgICAgICBlbHNlOgogICAgICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19QVVQoc2VsZik6CiAgICAgICAgc2VsZi5kb19IRUFEKCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX1BPU1Qoc2VsZik6CiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9jb3VudGVyCiAgICAgICAgZ2xvYmFsIGxpc3RfcmVxdWVzdF9tYXhfbnVtYmVyCgogICAgICAgIGlmIHNlbGYucGF0aC5zdGFydHN3aXRoKCIvcmVzZXRfY291bnRlcnMiKToKICAgICAgICAgICAgcXVlcnkgPSB1cmxsaWIucGFyc2UudXJscGFyc2Uoc2VsZi5wYXRoKS5xdWVyeQogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCgogICAgICAgICAgICBsaXN0X3JlcXVlc3RfY291bnRlciA9IDAKICAgICAgICAgICAgaWYgIm1heCIgaW4gcGFyYW1zOgogICAgICAgICAgICAgICAgbGlzdF9yZXF1ZXN0X21heF9udW1iZXIgPSBpbnQocGFyYW1zWyJtYXgiXVswXSkKCiAgICAgICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3BsYWluIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCiAgICAgICAgICAgIHJldHVybgoKICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19IRUFEKHNlbGYpOgogICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiKQogICAgICAgIGRhdGEgPSBzZWxmLnJmaWxlLnJlYWQoaW50KGNvbnRlbnRfbGVuZ3RoKSkgaWYgY29udGVudF9sZW5ndGggZWxzZSBOb25lCiAgICAgICAgciA9IHJlcXVlc3QoCiAgICAgICAgICAgIHNlbGYuY29tbWFuZCwKICAgICAgICAgICAgZiJodHRwOi8ve1VQU1RSRUFNX0hPU1R9e3NlbGYucGF0aH0iLAogICAgICAgICAgICBoZWFkZXJzPXNlbGYuaGVhZGVycywKICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2Uoci5zdGF0dXNfY29kZSkKICAgICAgICBmb3IgaywgdiBpbiByLmhlYWRlcnMuaXRlbXMoKToKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcihrLCB2KQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoci5jb250ZW50KQogICAgICAgIHNlbGYud2ZpbGUuY2xvc2UoKQoKCmNsYXNzIFRocmVhZGVkSFRUUFNlcnZlcihzb2NrZXRzZXJ2ZXIuVGhyZWFkaW5nTWl4SW4sIGh0dHAuc2VydmVyLkhUVFBTZXJ2ZXIpOgogICAgIiIiSGFuZGxlIHJlcXVlc3RzIGluIGEgc2VwYXJhdGUgdGhyZWFkLiIiIgoKCmh0dHBkID0gVGhyZWFkZWRIVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > no_list_objects.py'] (cluster.py:105, run_and_check)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:True nothrow:False cmd: ['python', 'no_list_objects.py', '8083'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:56 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:57 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8082/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:57 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8082/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:57 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:57 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:58 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:58 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:59 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:59 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:59 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:59 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:59 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8082/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:59 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8082/'] (cluster.py:105, run_and_check)
2026-10-27 17:29:59 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:29:59 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:29:59 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:00 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:01 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:01 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:01 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:01 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:01 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:01 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:01 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8082/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:01 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8082/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:03 [ 378 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2026-10-27 17:30:03 [ 378 ] DEBUG : echo.py answered OK on attempt 3 (mock_servers.py:53, start_mock_servers)
2026-10-27 17:30:03 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:03 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:05 [ 378 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2026-10-27 17:30:05 [ 378 ] DEBUG : no_list_objects.py answered OK on attempt 3 (mock_servers.py:53, start_mock_servers)
2026-10-27 17:30:06 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:06 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8080/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:06 [ 378 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2026-10-27 17:30:06 [ 378 ] DEBUG : mock_s3.py answered OK on attempt 4 (mock_servers.py:53, start_mock_servers)
2026-10-27 17:30:06 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:06 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:06 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:07 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:07 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:08 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:09 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:09 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:09 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:10 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:10 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:10 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:11 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:11 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:12 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:13 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:13 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:13 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:14 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:14 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:14 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:15 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:15 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:15 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:16 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:16 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:17 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:18 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:18 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:18 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:19 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:19 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:19 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:20 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:20 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:20 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:21 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:21 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:22 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:23 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:23 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:23 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:24 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:24 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:24 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:25 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:25 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:26 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:27 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:27 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:27 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:28 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:28 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:28 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:29 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:29 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:29 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:30 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:30 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:31 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:32 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:32 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:32 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:33 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:33 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:33 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:34 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:34 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:35 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:36 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:36 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:36 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:37 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:37 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:37 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:38 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:38 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:38 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:39 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:39 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:40 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:41 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:41 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:41 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:42 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:42 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:42 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:43 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:43 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:43 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:44 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:44 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:45 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:46 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:46 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:46 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:47 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:47 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:47 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:48 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:48 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:49 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:50 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:50 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:50 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:51 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:51 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:51 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:52 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:52 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:52 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:53 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:53 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:54 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:55 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:55 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:55 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:56 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:56 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:56 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:57 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:57 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:58 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:30:59 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:30:59 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:30:59 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:00 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:00 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:00 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:01 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:01 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:01 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:02 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:02 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:03 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:04 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:04 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:04 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:05 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:05 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:05 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:06 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:06 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:06 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:07 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:07 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:08 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:09 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:09 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:09 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:10 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:10 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:10 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:11 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:11 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:11 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:12 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:12 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:13 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:14 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:14 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:14 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:15 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:15 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:15 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:16 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:16 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:17 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:18 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:18 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:18 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:19 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:19 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:19 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:20 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:20 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:20 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:21 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:21 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:22 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:23 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:23 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:23 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:24 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:24 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:24 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:25 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:25 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:26 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:27 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:27 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:27 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:28 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:28 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:28 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:29 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:29 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:29 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:30 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:30 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:31 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:32 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:32 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:32 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:33 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:33 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:33 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:34 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:34 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:35 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:36 [ 378 ] DEBUG : run container_id:rootteststorages3invalidenvcredentials_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2004, exec_in_container)
2026-10-27 17:31:36 [ 378 ] DEBUG : Command:['docker', 'exec', 'rootteststorages3invalidenvcredentials_resolver_1', 'curl', '-s', 'http://localhost:8081/'] (cluster.py:105, run_and_check)
2026-10-27 17:31:36 [ 378 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2026-10-27 17:31:36 [ 378 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', '--project-name', 'rootteststorages3invalidenvcredentials', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_minio1_1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_resolver_1                                ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_proxy2_1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_proxy1_1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_minio1_1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_resolver_1                                ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_proxy2_1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Stopping rootteststorages3invalidenvcredentials_proxy1_1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/logs/stderr.log* || true'] (cluster.py:105, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/.env', '--project-name', 'rootteststorages3invalidenvcredentials', '--file', '/ClickHouse/tests/integration/test_storage_s3/_instances_invalid_env_credentials_0/s3_with_invalid_environment_credentials/docker-compose.yml', '--file', '/compose/docker_compose_minio.yml', 'down', '--volumes'] (cluster.py:105, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1 ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_minio1_1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_resolver_1                                ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_proxy2_1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_proxy1_1                                  ...  (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_proxy2_1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_s3_with_invalid_environment_credentials_1 ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_proxy1_1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_resolver_1                                ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing rootteststorages3invalidenvcredentials_minio1_1                                  ... done (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing network rootteststorages3invalidenvcredentials_default (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Stderr:Removing volume rootteststorages3invalidenvcredentials_data1-1 (cluster.py:115, run_and_check)
2026-10-27 17:31:58 [ 378 ] DEBUG : Cleanup called (cluster.py:754, cleanup)
2026-10-27 17:31:58 [ 378 ] DEBUG : !!! Docker info: Client: Docker Engine - Community
 Version:    29.9.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 7
  Running: 6
  Paused: 1
  Stopped: 0
 Images: 35
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b7a25261e029d68978bbf290df0a62fc819b3c51
 runc version: v1.5.2-0-g29dd3dc2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-130-generic
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.59GiB
 Name: acf8b56b85a3
 ID: 38ee18d9-7524-4ca7-9a78-b5a339417f66
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  65.108.242.32:5000
  127.0.0.0/8
 Registry Mirrors:
  http://65.108.242.32:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 24

 (cluster.py:722, print_all_docker_pieces)
2026-10-27 17:31:58 [ 378 ] DEBUG : Docker networks for project rootteststorages3invalidenvcredentials are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:733, print_all_docker_pieces)
2026-10-27 17:31:59 [ 378 ] DEBUG : Docker containers for project rootteststorages3invalidenvcredentials are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:741, print_all_docker_pieces)
2026-10-27 17:31:59 [ 378 ] DEBUG : Docker volumes for project rootteststorages3invalidenvcredentials are DRIVER    VOLUME NAME
 (cluster.py:749, print_all_docker_pieces)
2026-10-27 17:31:59 [ 378 ] DEBUG : Command:docker container list --all --filter name='^/rootteststorages3invalidenvcredentials_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check)
2026-10-27 17:31:59 [ 378 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup)
2026-10-27 17:31:59 [ 378 ] DEBUG : No running containers for project: rootteststorages3invalidenvcredentials (cluster.py:782, cleanup)
2026-10-27 17:31:59 [ 378 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup)
2026-10-27 17:31:59 [ 378 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup)
2026-10-27 17:31:59 [ 378 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check)
2026-10-27 17:31:59 [ 378 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check)
2026-10-27 17:31:59 [ 378 ] DEBUG : Images pruned (cluster.py:807, cleanup)
2026-10-27 17:31:59 [ 378 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup)
2026-10-27 17:31:59 [ 378 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check)
2026-10-27 17:31:59 [ 378 ] DEBUG : Stdout:3 (cluster.py:113, run_and_check)
_________ ERROR at setup of test_with_invalid_environment_credentials __________
[gw7] linux -- Python 3.10.12 /usr/bin/python3

    @pytest.fixture(scope="module")
    def started_cluster():
        try:
            cluster = ClickHouseCluster(__file__)
            cluster.add_instance(
                "s3_with_invalid_environment_credentials",
                with_minio=True,
                env_variables={
                    "AWS_ACCESS_KEY_ID": "aws",
                    "AWS_SECRET_ACCESS_KEY": "aws123",
                },
                main_configs=[
                    "configs/use_environment_credentials.xml",
                    "configs/named_collections.xml",
                ],
                user_configs=["configs/users.xml"],
            )
    
            logging.info("Starting cluster...")
            cluster.start()
            logging.info("Cluster started")
    
            prepare_s3_bucket(cluster)
            logging.info("S3 bucket created")
>           run_s3_mocks(cluster)

test_storage_s3/test_invalid_env_credentials.py:104: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_storage_s3/test_invalid_env_credentials.py:68: in run_s3_mocks
    start_mock_servers(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cluster = <helpers.cluster.ClickHouseCluster object at 0x7fd6cb3c7eb0>
script_dir = '/ClickHouse/tests/integration/test_storage_s3/s3_mocks'
mocks = [('mock_s3.py', 'resolver', '8080'), ('unstable_server.py', 'resolver', '8081'), ('echo.py', 'resolver', '8082'), ('no_list_objects.py', 'resolver', '8083')]
timeout = 100

    def start_mock_servers(cluster, script_dir, mocks, timeout=100):
        server_names = [mock[0] for mock in mocks]
        server_names_with_desc = (
            f"{'server' if len(server_names) == 1 else 'servers'} {','.join(server_names)}"
        )
        logging.info(f"Starting mock {server_names_with_desc}")
    
        start_time = time.time()
        mocks_to_check = {}
    
        for server_name, container, port in mocks:
            filepath = os.path.join(script_dir, server_name)
            container_id = cluster.get_container_id(container)
            mocks_to_check[server_name] = (container_id, port)
    
            cluster.copy_file_to_container(
                container_id,
                filepath,
                server_name,
            )
    
            cluster.exec_in_container(
                container_id,
                ["python", server_name, str(port)],
                detach=True,
            )
    
        # Wait for the server to start.
        attempt = 1
        while mocks_to_check:
            for server_name in list(mocks_to_check.keys()):
                container_id, port = mocks_to_check[server_name]
    
                ping_response = cluster.exec_in_container(
                    container_id,
                    ["curl", "-s", f"http://localhost:{port}/"],
                    nothrow=True,
                )
    
                if ping_response == "OK":
                    logging.debug(
                        f"{server_name} answered {ping_response} on attempt {attempt}"
                    )
                    del mocks_to_check[server_name]
                elif time.time() - start_time > timeout:
                    assert (
>                       ping_response == "OK"
                    ), 'Expected "OK", but got "{}" from {}'.format(
                        ping_response, server_name
                    )
E                   AssertionError: Expected "OK", but got "" from unstable_server.py

helpers/mock_servers.py:59: AssertionError
============================== slowest durations ===============================
149.37s call     test_storage_azure_blob_storage/test.py::test_schema_inference_cache
141.46s setup    test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections
97.13s setup    test_storage_hdfs/test.py::test_bad_hdfs_uri
77.90s call     test_storage_kerberized_hdfs/test.py::test_read_table_expired
71.34s setup    test_storage_azure_blob_storage/test.py::test_partition_by_const_column
56.99s setup    test_storage_kerberized_hdfs/test.py::test_cache_path
44.66s teardown test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired
32.87s setup    test_zero_copy_fetch/test.py::test_concurrent_move_to_s3
30.44s setup    test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part
23.88s call     test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired
22.79s teardown test_zero_copy_fetch/test.py::test_zero_copy_mutation
22.77s teardown test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part
22.42s teardown test_storage_hdfs/test.py::test_write_table
20.91s setup    test_zookeeper_config/test_password.py::test_identity
20.79s call     test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs
19.37s call     test_storage_hdfs/test.py::test_schema_inference_cache
19.06s setup    test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric
16.32s call     test_storage_kerberized_hdfs/test.py::test_read_table
15.83s call     test_storage_azure_blob_storage/test.py::test_put_get_with_globs
15.61s call     test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf
15.02s call     test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf
14.44s setup    test_storage_url/test.py::test_file_formats[CSV]
13.66s teardown test_system_metrics/test.py::test_metrics_storage_buffer_size
12.99s setup    test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance
12.69s setup    test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name
8.98s call     test_storage_kerberized_hdfs/test.py::test_read_write_storage
8.31s call     test_storage_hdfs/test.py::test_multiple_inserts
8.20s call     test_zero_copy_fetch/test.py::test_concurrent_move_to_s3
7.86s teardown test_storage_url/test.py::test_url_cluster_with_named_collection
6.99s teardown test_zookeeper_config/test_password.py::test_identity
6.78s call     test_storage_hdfs/test.py::test_seekable_formats
6.00s call     test_storage_hdfs/test.py::test_globs_in_read_table
5.60s teardown test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance
5.19s teardown test_wrong_db_or_table_name/test.py::test_wrong_table_name
4.61s call     test_storage_hdfs/test.py::test_schema_inference
4.33s teardown test_storage_azure_blob_storage/test.py::test_truncate
3.06s call     test_system_metrics/test.py::test_broken_tables_readonly_metric
2.63s call     test_storage_hdfs/test.py::test_schema_inference_with_globs
2.29s call     test_zero_copy_fetch/test.py::test_zero_copy_mutation
1.74s call     test_storage_kerberized_hdfs/test.py::test_prohibited
1.57s call     test_storage_hdfs/test.py::test_cluster_join
1.52s call     test_storage_hdfs/test.py::test_truncate_table
1.50s call     test_storage_hdfs/test.py::test_read_subcolumns
1.45s call     test_storage_kerberized_hdfs/test.py::test_two_users
1.39s call     test_storage_hdfs/test.py::test_cluster_macro
1.32s call     test_storage_hdfs/test.py::test_virtual_columns_2
1.21s call     test_storage_url/test.py::test_file_formats[Parquet]
1.06s call     test_zero_copy_fetch/test.py::test_fetch_correct_volume
0.96s call     test_storage_hdfs/test.py::test_skip_empty_files
0.94s call     test_storage_hdfs/test.py::test_read_write_storage_with_globs
0.90s call     test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part
0.89s call     test_storage_hdfs/test.py::test_partition_by
0.87s call     test_storage_url/test.py::test_file_formats[TSV]
0.86s call     test_system_metrics/test.py::test_metrics_storage_buffer_size
0.83s call     test_storage_hdfs/test.py::test_overwrite
0.78s call     test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name
0.77s call     test_storage_hdfs/test.py::test_write_table
0.75s call     test_storage_hdfs/test.py::test_format_detection
0.75s call     test_storage_hdfs/test.py::test_read_write_storage
0.73s call     test_storage_hdfs/test.py::test_read_files_with_spaces
0.73s call     test_storage_url/test.py::test_table_function_url_access_rights
0.72s call     test_storage_hdfs/test.py::test_write_gz_storage
0.69s call     test_storage_azure_blob_storage/test.py::test_read_subcolumns
0.68s call     test_storage_hdfs/test.py::test_write_gzip_storage
0.68s call     test_storage_hdfs/test.py::test_hdfs_directory_not_exist
0.65s call     test_storage_hdfs/test.py::test_insert_select_schema_inference
0.60s call     test_storage_url/test.py::test_url_cluster
0.59s call     test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric
0.56s call     test_storage_hdfs/test.py::test_read_write_table
0.51s call     test_storage_url/test.py::test_file_formats[JSONEachRow]
0.50s call     test_storage_url/test.py::test_partition_by
0.50s call     test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs
0.44s call     test_storage_url/test.py::test_url_cluster_with_named_collection
0.43s call     test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name
0.41s call     test_storage_hdfs/test.py::test_hdfsCluster
0.37s call     test_storage_azure_blob_storage/test.py::test_simple_read_write
0.36s call     test_storage_url/test.py::test_file_formats[CSV]
0.34s call     test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]
0.32s call     test_storage_azure_blob_storage/test.py::test_partition_by_string_column
0.30s call     test_zookeeper_config/test_password.py::test_identity
0.28s call     test_storage_hdfs/test.py::test_storage_with_multidirectory_glob
0.28s call     test_wrong_db_or_table_name/test.py::test_wrong_database_name
0.28s call     test_wrong_db_or_table_name/test.py::test_wrong_table_name
0.27s call     test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]
0.27s call     test_storage_azure_blob_storage/test.py::test_truncate
0.27s call     test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function
0.26s call     test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1
0.25s call     test_storage_azure_blob_storage/test.py::test_partition_by_const_column
0.23s call     test_storage_hdfs/test.py::test_virtual_columns
0.23s call     test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf
0.22s call     test_storage_hdfs/test.py::test_bad_hdfs_uri
0.21s call     test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards
0.20s call     test_storage_azure_blob_storage/test.py::test_simple_write_connection_string
0.20s call     test_storage_azure_blob_storage/test.py::test_simple_write_account_string
0.20s call     test_storage_hdfs/test.py::test_read_table_with_default
0.20s call     test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2
0.18s call     test_storage_kerberized_hdfs/test.py::test_cache_path
0.17s call     test_storage_hdfs/test.py::test_read_write_gzip_table
0.17s call     test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz
0.17s call     test_storage_hdfs/test.py::test_read_write_table_with_parameter_none
0.15s call     test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards
0.15s call     test_storage_azure_blob_storage/test.py::test_partition_by_tf
0.14s call     test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function
0.13s call     test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function
0.13s call     test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function
0.11s call     test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip
0.07s call     test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container
0.03s call     test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance
0.00s teardown test_storage_hdfs/test.py::test_bad_hdfs_uri
0.00s call     test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance
0.00s teardown test_storage_kerberized_hdfs/test.py::test_cache_path
0.00s teardown test_zero_copy_fetch/test.py::test_concurrent_move_to_s3
0.00s setup    test_storage_hdfs/test.py::test_truncate_table
0.00s teardown test_storage_kerberized_hdfs/test.py::test_read_write_storage
0.00s setup    test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]
0.00s teardown test_storage_kerberized_hdfs/test.py::test_read_table
0.00s teardown test_storage_kerberized_hdfs/test.py::test_read_table_expired
0.00s setup    test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]
0.00s teardown test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections
0.00s setup    test_storage_hdfs/test.py::test_overwrite
0.00s teardown test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric
0.00s setup    test_storage_hdfs/test.py::test_virtual_columns
0.00s setup    test_storage_url/test.py::test_partition_by
0.00s setup    test_storage_kerberized_hdfs/test.py::test_two_users
0.00s setup    test_wrong_db_or_table_name/test.py::test_wrong_database_name
0.00s setup    test_storage_hdfs/test.py::test_schema_inference
0.00s teardown test_storage_url/test.py::test_file_formats[TSV]
0.00s setup    test_storage_url/test.py::test_file_formats[TSV]
0.00s teardown test_storage_hdfs/test.py::test_multiple_inserts
0.00s setup    test_storage_hdfs/test.py::test_cluster_macro
0.00s setup    test_zero_copy_fetch/test.py::test_zero_copy_mutation
0.00s setup    test_storage_hdfs/test.py::test_multiple_inserts
0.00s setup    test_storage_azure_blob_storage/test.py::test_schema_inference_cache
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function
0.00s setup    test_storage_hdfs/test.py::test_cluster_join
0.00s setup    test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz
0.00s teardown test_storage_azure_blob_storage/test.py::test_partition_by_const_column
0.00s teardown test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs
0.00s teardown test_storage_hdfs/test.py::test_truncate_table
0.00s teardown test_storage_hdfs/test.py::test_insert_select_schema_inference
0.00s teardown test_zero_copy_fetch/test.py::test_fetch_correct_volume
0.00s setup    test_storage_kerberized_hdfs/test.py::test_prohibited
0.00s setup    test_storage_hdfs/test.py::test_read_write_gzip_table
0.00s setup    test_storage_hdfs/test.py::test_write_gzip_storage
0.00s setup    test_zero_copy_fetch/test.py::test_fetch_correct_volume
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_connection_string
0.00s setup    test_storage_azure_blob_storage/test.py::test_partition_by_string_column
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function
0.00s teardown test_storage_azure_blob_storage/test.py::test_put_get_with_globs
0.00s setup    test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired
0.00s setup    test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container
0.00s teardown test_storage_hdfs/test.py::test_storage_with_multidirectory_glob
0.00s setup    test_storage_hdfs/test.py::test_seekable_formats
0.00s teardown test_storage_azure_blob_storage/test.py::test_read_subcolumns
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_connection_string
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function
0.00s teardown test_storage_hdfs/test.py::test_virtual_columns
0.00s setup    test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2
0.00s teardown test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf
0.00s teardown test_storage_hdfs/test.py::test_read_write_table
0.00s setup    test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf
0.00s setup    test_storage_hdfs/test.py::test_virtual_columns_2
0.00s teardown test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container
0.00s setup    test_storage_kerberized_hdfs/test.py::test_read_write_storage
0.00s setup    test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf
0.00s teardown test_storage_hdfs/test.py::test_read_write_table_with_parameter_none
0.00s setup    test_storage_hdfs/test.py::test_globs_in_read_table
0.00s teardown test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf
0.00s teardown test_storage_azure_blob_storage/test.py::test_schema_inference_cache
0.00s setup    test_storage_kerberized_hdfs/test.py::test_read_table_expired
0.00s setup    test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1
0.00s setup    test_wrong_db_or_table_name/test.py::test_wrong_table_name
0.00s setup    test_storage_azure_blob_storage/test.py::test_read_subcolumns
0.00s teardown test_storage_hdfs/test.py::test_read_files_with_spaces
0.00s setup    test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards
0.00s setup    test_storage_hdfs/test.py::test_hdfs_directory_not_exist
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_read_write
0.00s teardown test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_read_write
0.00s teardown test_storage_hdfs/test.py::test_read_table_with_default
0.00s setup    test_storage_hdfs/test.py::test_insert_select_schema_inference
0.00s setup    test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf
0.00s setup    test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_account_string
0.00s setup    test_storage_hdfs/test.py::test_read_write_table_with_parameter_none
0.00s setup    test_storage_hdfs/test.py::test_skip_empty_files
0.00s teardown test_storage_hdfs/test.py::test_write_gz_storage
0.00s setup    test_storage_hdfs/test.py::test_read_subcolumns
0.00s teardown test_storage_hdfs/test.py::test_hdfsCluster
0.00s teardown test_storage_url/test.py::test_file_formats[CSV]
0.00s teardown test_storage_hdfs/test.py::test_read_write_gzip_table
0.00s teardown test_storage_url/test.py::test_file_formats[Parquet]
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2
0.00s teardown test_storage_hdfs/test.py::test_format_detection
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1
0.00s teardown test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]
0.00s setup    test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function
0.00s setup    test_storage_hdfs/test.py::test_format_detection
0.00s teardown test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name
0.00s setup    test_storage_hdfs/test.py::test_storage_with_multidirectory_glob
0.00s teardown test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards
0.00s teardown test_storage_hdfs/test.py::test_seekable_formats
0.00s teardown test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards
0.00s teardown test_storage_hdfs/test.py::test_schema_inference_with_globs
0.00s teardown test_storage_hdfs/test.py::test_cluster_join
0.00s teardown test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf
0.00s teardown test_storage_kerberized_hdfs/test.py::test_two_users
0.00s teardown test_storage_hdfs/test.py::test_hdfs_directory_not_exist
0.00s teardown test_wrong_db_or_table_name/test.py::test_wrong_database_name
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_account_string
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function
0.00s teardown test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]
0.00s teardown test_storage_hdfs/test.py::test_cluster_macro
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function
0.00s teardown test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function
0.00s setup    test_storage_url/test.py::test_file_formats[JSONEachRow]
0.00s teardown test_storage_hdfs/test.py::test_skip_empty_files
0.00s setup    test_storage_hdfs/test.py::test_read_write_storage_with_globs
0.00s setup    test_storage_hdfs/test.py::test_read_write_table
0.00s setup    test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance
0.00s setup    test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name
0.00s teardown test_system_metrics/test.py::test_broken_tables_readonly_metric
0.00s setup    test_system_metrics/test.py::test_metrics_storage_buffer_size
0.00s setup    test_storage_azure_blob_storage/test.py::test_truncate
0.00s setup    test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials
0.00s setup    test_system_metrics/test.py::test_broken_tables_readonly_metric
0.00s setup    test_storage_hdfs/test.py::test_partition_by
0.00s setup    test_storage_hdfs/test.py::test_read_files_with_spaces
0.00s setup    test_storage_hdfs/test.py::test_read_write_storage
0.00s teardown test_storage_hdfs/test.py::test_read_write_storage_with_globs
0.00s setup    test_storage_url/test.py::test_table_function_url_access_rights
0.00s teardown test_storage_hdfs/test.py::test_globs_in_read_table
0.00s teardown test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name
0.00s setup    test_storage_azure_blob_storage/test.py::test_put_get_with_globs
0.00s teardown test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance
0.00s teardown test_storage_hdfs/test.py::test_partition_by
0.00s setup    test_storage_url/test.py::test_url_cluster_with_named_collection
0.00s teardown test_storage_hdfs/test.py::test_overwrite
0.00s setup    test_storage_azure_blob_storage/test.py::test_partition_by_tf
0.00s teardown test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials
0.00s teardown test_storage_hdfs/test.py::test_read_write_storage
0.00s setup    test_storage_url/test.py::test_file_formats[Parquet]
0.00s teardown test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip
0.00s teardown test_storage_hdfs/test.py::test_schema_inference_cache
0.00s setup    test_storage_kerberized_hdfs/test.py::test_read_table
0.00s setup    test_storage_hdfs/test.py::test_write_gz_storage
0.00s teardown test_storage_url/test.py::test_partition_by
0.00s setup    test_storage_hdfs/test.py::test_hdfsCluster
0.00s teardown test_storage_hdfs/test.py::test_read_subcolumns
0.00s teardown test_storage_azure_blob_storage/test.py::test_partition_by_tf
0.00s setup    test_storage_hdfs/test.py::test_read_table_with_default
0.00s setup    test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip
0.00s teardown test_storage_hdfs/test.py::test_write_gzip_storage
0.00s teardown test_storage_hdfs/test.py::test_virtual_columns_2
0.00s teardown test_storage_kerberized_hdfs/test.py::test_prohibited
0.00s setup    test_storage_hdfs/test.py::test_schema_inference_cache
0.00s teardown test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz
0.00s setup    test_storage_hdfs/test.py::test_schema_inference_with_globs
0.00s setup    test_storage_url/test.py::test_url_cluster
0.00s setup    test_storage_hdfs/test.py::test_write_table
0.00s teardown test_storage_hdfs/test.py::test_schema_inference
0.00s teardown test_storage_url/test.py::test_file_formats[JSONEachRow]
0.00s teardown test_storage_azure_blob_storage/test.py::test_partition_by_string_column
0.00s teardown test_storage_url/test.py::test_table_function_url_access_rights
0.00s teardown test_storage_url/test.py::test_url_cluster
=========================== short test summary info ============================
ERROR test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections
ERROR test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials
PASSED test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance
PASSED test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance
PASSED test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name
PASSED test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name
PASSED test_wrong_db_or_table_name/test.py::test_wrong_database_name
PASSED test_wrong_db_or_table_name/test.py::test_wrong_table_name
PASSED test_storage_url/test.py::test_file_formats[CSV]
PASSED test_storage_url/test.py::test_file_formats[JSONEachRow]
PASSED test_storage_url/test.py::test_file_formats[Parquet]
PASSED test_storage_url/test.py::test_file_formats[TSV]
PASSED test_storage_url/test.py::test_partition_by
PASSED test_storage_url/test.py::test_table_function_url_access_rights
PASSED test_storage_url/test.py::test_url_cluster
PASSED test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric
PASSED test_storage_url/test.py::test_url_cluster_with_named_collection
PASSED test_system_metrics/test.py::test_broken_tables_readonly_metric
PASSED test_system_metrics/test.py::test_metrics_storage_buffer_size
PASSED test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part
PASSED test_zero_copy_fetch/test.py::test_concurrent_move_to_s3
PASSED test_zero_copy_fetch/test.py::test_fetch_correct_volume
PASSED test_zero_copy_fetch/test.py::test_zero_copy_mutation
PASSED test_storage_kerberized_hdfs/test.py::test_cache_path
PASSED test_storage_kerberized_hdfs/test.py::test_prohibited
PASSED test_storage_azure_blob_storage/test.py::test_partition_by_const_column
PASSED test_storage_azure_blob_storage/test.py::test_partition_by_string_column
PASSED test_storage_azure_blob_storage/test.py::test_partition_by_tf
PASSED test_storage_kerberized_hdfs/test.py::test_read_table
PASSED test_storage_azure_blob_storage/test.py::test_put_get_with_globs
PASSED test_storage_hdfs/test.py::test_bad_hdfs_uri
PASSED test_storage_hdfs/test.py::test_cluster_join
PASSED test_storage_hdfs/test.py::test_cluster_macro
PASSED test_storage_hdfs/test.py::test_format_detection
PASSED test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf
PASSED test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container
PASSED test_storage_azure_blob_storage/test.py::test_read_subcolumns
PASSED test_storage_hdfs/test.py::test_globs_in_read_table
PASSED test_storage_hdfs/test.py::test_hdfsCluster
PASSED test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards
PASSED test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards
PASSED test_storage_hdfs/test.py::test_hdfs_directory_not_exist
PASSED test_storage_hdfs/test.py::test_insert_select_schema_inference
PASSED test_storage_hdfs/test.py::test_multiple_inserts
PASSED test_storage_hdfs/test.py::test_overwrite
PASSED test_storage_hdfs/test.py::test_partition_by
PASSED test_storage_hdfs/test.py::test_read_files_with_spaces
PASSED test_storage_hdfs/test.py::test_read_subcolumns
PASSED test_storage_hdfs/test.py::test_read_table_with_default
PASSED test_storage_hdfs/test.py::test_read_write_gzip_table
PASSED test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz
PASSED test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip
PASSED test_storage_hdfs/test.py::test_read_write_storage
PASSED test_storage_hdfs/test.py::test_read_write_storage_with_globs
PASSED test_storage_hdfs/test.py::test_read_write_table
PASSED test_storage_hdfs/test.py::test_read_write_table_with_parameter_none
PASSED test_storage_hdfs/test.py::test_schema_inference
PASSED test_storage_hdfs/test.py::test_schema_inference_cache
PASSED test_storage_hdfs/test.py::test_schema_inference_with_globs
PASSED test_storage_kerberized_hdfs/test.py::test_read_table_expired
PASSED test_storage_hdfs/test.py::test_seekable_formats
PASSED test_storage_hdfs/test.py::test_skip_empty_files
PASSED test_storage_hdfs/test.py::test_storage_with_multidirectory_glob
PASSED test_storage_hdfs/test.py::test_truncate_table
PASSED test_storage_hdfs/test.py::test_virtual_columns
PASSED test_storage_kerberized_hdfs/test.py::test_read_write_storage
PASSED test_storage_hdfs/test.py::test_virtual_columns_2
PASSED test_zookeeper_config/test_password.py::test_identity
PASSED test_storage_hdfs/test.py::test_write_gz_storage
PASSED test_storage_kerberized_hdfs/test.py::test_two_users
PASSED test_storage_hdfs/test.py::test_write_gzip_storage
PASSED test_storage_hdfs/test.py::test_write_table
PASSED test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired
PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_cache
PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs
PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf
PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs
PASSED test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf
PASSED test_storage_azure_blob_storage/test.py::test_simple_read_write
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_account_string
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_connection_string
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2
PASSED test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function
PASSED test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]
PASSED test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]
PASSED test_storage_azure_blob_storage/test.py::test_truncate
=================== 88 passed, 2 errors in 298.66s (0:04:58) ===================
Traceback (most recent call last):
  File "/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration/./runner", line 455, in <module>
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run   --rm --name clickhouse_integration_tests_wa6pwn --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=1864 -e DOCKER_BASE_TAG=1864 -e DOCKER_KERBERIZED_HADOOP_TAG=1864 -e DOCKER_KERBEROS_KDC_TAG=1864 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=1864 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=1864 -e DOCKER_MYSQL_JS_CLIENT_TAG=1864 -e DOCKER_MYSQL_PHP_CLIENT_TAG=1864 -e DOCKER_NGINX_DAV_TAG=1864 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=1864   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600  -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_storage_azure_blob_storage/test.py::test_partition_by_const_column test_storage_azure_blob_storage/test.py::test_partition_by_string_column test_storage_azure_blob_storage/test.py::test_partition_by_tf test_storage_azure_blob_storage/test.py::test_put_get_with_globs test_storage_azure_blob_storage/test.py::test_put_get_with_globs_tf test_storage_azure_blob_storage/test.py::test_read_from_not_existing_container test_storage_azure_blob_storage/test.py::test_read_subcolumns test_storage_azure_blob_storage/test.py::test_schema_inference_cache test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs test_storage_azure_blob_storage/test.py::test_schema_inference_from_globs_tf test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs test_storage_azure_blob_storage/test.py::test_schema_inference_no_globs_tf test_storage_azure_blob_storage/test.py::test_simple_read_write test_storage_azure_blob_storage/test.py::test_simple_write_account_string test_storage_azure_blob_storage/test.py::test_simple_write_account_string_table_function test_storage_azure_blob_storage/test.py::test_simple_write_connection_string test_storage_azure_blob_storage/test.py::test_simple_write_connection_string_table_function test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1 test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_1_table_function test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2 test_storage_azure_blob_storage/test.py::test_simple_write_named_collection_2_table_function 'test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[bin]' 'test_storage_azure_blob_storage/test.py::test_storage_azure_get_gzip[gz]' test_storage_azure_blob_storage/test.py::test_truncate test_storage_hdfs/test.py::test_bad_hdfs_uri test_storage_hdfs/test.py::test_cluster_join test_storage_hdfs/test.py::test_cluster_macro test_storage_hdfs/test.py::test_format_detection test_storage_hdfs/test.py::test_globs_in_read_table test_storage_hdfs/test.py::test_hdfsCluster test_storage_hdfs/test.py::test_hdfsCluster_skip_unavailable_shards test_storage_hdfs/test.py::test_hdfsCluster_unset_skip_unavailable_shards test_storage_hdfs/test.py::test_hdfs_directory_not_exist test_storage_hdfs/test.py::test_insert_select_schema_inference test_storage_hdfs/test.py::test_multiple_inserts test_storage_hdfs/test.py::test_overwrite test_storage_hdfs/test.py::test_partition_by test_storage_hdfs/test.py::test_read_files_with_spaces test_storage_hdfs/test.py::test_read_subcolumns test_storage_hdfs/test.py::test_read_table_with_default test_storage_hdfs/test.py::test_read_write_gzip_table test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_auto_gz test_storage_hdfs/test.py::test_read_write_gzip_table_with_parameter_gzip test_storage_hdfs/test.py::test_read_write_storage test_storage_hdfs/test.py::test_read_write_storage_with_globs test_storage_hdfs/test.py::test_read_write_table test_storage_hdfs/test.py::test_read_write_table_with_parameter_none test_storage_hdfs/test.py::test_schema_inference test_storage_hdfs/test.py::test_schema_inference_cache test_storage_hdfs/test.py::test_schema_inference_with_globs test_storage_hdfs/test.py::test_seekable_formats test_storage_hdfs/test.py::test_skip_empty_files test_storage_hdfs/test.py::test_storage_with_multidirectory_glob test_storage_hdfs/test.py::test_truncate_table test_storage_hdfs/test.py::test_virtual_columns test_storage_hdfs/test.py::test_virtual_columns_2 test_storage_hdfs/test.py::test_write_gz_storage test_storage_hdfs/test.py::test_write_gzip_storage test_storage_hdfs/test.py::test_write_table test_storage_kerberized_hdfs/test.py::test_cache_path test_storage_kerberized_hdfs/test.py::test_prohibited test_storage_kerberized_hdfs/test.py::test_read_table test_storage_kerberized_hdfs/test.py::test_read_table_expired test_storage_kerberized_hdfs/test.py::test_read_write_storage test_storage_kerberized_hdfs/test.py::test_two_users test_storage_kerberized_hdfs/test.py::test_write_storage_not_expired test_storage_s3/test_invalid_env_credentials.py::test_no_sign_named_collections test_storage_s3/test_invalid_env_credentials.py::test_with_invalid_environment_credentials 'test_storage_url/test.py::test_file_formats[CSV]' 'test_storage_url/test.py::test_file_formats[JSONEachRow]' 'test_storage_url/test.py::test_file_formats[Parquet]' 'test_storage_url/test.py::test_file_formats[TSV]' test_storage_url/test.py::test_partition_by test_storage_url/test.py::test_table_function_url_access_rights test_storage_url/test.py::test_url_cluster test_storage_url/test.py::test_url_cluster_with_named_collection test_system_metrics/test.py::test_attach_without_zk_incr_readonly_metric test_system_metrics/test.py::test_broken_tables_readonly_metric test_system_metrics/test.py::test_metrics_storage_buffer_size test_tcp_handler_http_responses/test_case.py::test_request_to_http_full_instance test_tcp_handler_http_responses/test_case.py::test_request_to_http_less_instance test_wrong_db_or_table_name/test.py::test_drop_wrong_database_name test_wrong_db_or_table_name/test.py::test_drop_wrong_table_name test_wrong_db_or_table_name/test.py::test_wrong_database_name test_wrong_db_or_table_name/test.py::test_wrong_table_name test_zero_copy_fetch/test.py::test_concurrent_move_to_s3 test_zero_copy_fetch/test.py::test_fetch_correct_volume test_zero_copy_fetch/test.py::test_zero_copy_mutation test_zero_copy_replication_drop_detached_part/test.py::test_drop_detached_part test_zookeeper_config/test_password.py::test_identity  -vvv" altinityinfra/integration-tests-runner:1864 ' returned non-zero exit status 1.